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ABSTRACT 


The  source  selection  phase  in  government  acquisitions  is  so  complicated  in  nature 
because  it  involves  multi-criteria  decision  making  that  is  supposed  to  respond  to  various 
requirements  and  subjectivity  is  usually  inevitable  in  this  kind  of  a  decision  making 
process.  The  purpose  of  this  project  is  to  demonstrate  how  the  USAF’s  current  source 
selection  method  (color  rating  method)  is  incompetent  in  showing  small  differences 
between  proposed  products,  how  this  inadequacy  leads  to  subjective  decisions,  and  that 
the  use  of  information  technology  tools  can  augment  objectivity  in  this  process. 

In  this  study,  USAF’s  KC-X  Tanker  Replacement  Program  has  been  selected  as 
the  program  to  be  used  to  frame  the  research  questions.  Two  models  with  two  versions 
built  on  Microsoft  Excel  spreadsheets  using  publicly  available  KC-X  program  data  are 
used  to  compare  the  USAF’s  color  rating  method  and  weighted  sum  method,  which  is  a 
multi-criteria  decision  making  tool.  It  is  presented  that  if  the  USAF  had  used  the 
weighted  sum  method  as  its  evaluation  method,  the  winner  of  the  KC-X  program  could 
have  been  different.  The  findings  prove  that  the  color  rating  method  is  not  capable  of 
reflecting  small  differences  and  information  technology  tools  can  help  decision  makers 
choose  the  best  value  offeror  with  less  subjectivity. 
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I.  INTRODUCTION 


The  source  selection  phase  of  government  acquisitions  involves  many  interrelated 
tasks.  At  the  beginning,  teams  of  experts  determine  the  evaluation  factors  and  significant 
subfactors  that  represent  the  key  concerns  of  the  procurement  and  incorporate  those 
factors  into  a  fonnal  document  which  is  called  a  Request  for  Proposal  (RFP).  Once 
proposals  are  received,  experts  evaluate  each  proposal  against  the  standards  provided  in 
the  RFP.  If  negotiated  contracting  is  the  selected  method  for  the  acquisition,  contractors 
are  allowed  to  revise  their  offers  and  discussions  are  conducted  before  the  contract  is 
awarded.  Otherwise,  evaluations  are  conducted  without  negotiations  and  the  contract  is 
awarded  directly  by  the  Source  Selection  Authority  (SSA)  based  on  the  evaluations.1 

Even  though  objective  factors  and  subfactors  are  used  in  the  source  selection 
phase,  subjectivity  is  inevitable  in  government  contracting.  While  the  Federal  Acquisition 
Regulation  (FAR)  does  not  provide  any  clear  guidance  on  evaluating  proposals,  the 
source  selection  decision  is  usually  made  based  on  some  evaluations  “conducted  using 
any  rating  method  or  a  combination  of  methods,  including  color  or  adjectival  ratings, 
numerical  weights,  and  ordinal  rankings.”2  Besides,  proposals  are  evaluated  by  different 
personnel  and  conclusions  may  differ  significantly,  even  for  the  same  factor.3  Therefore, 
considering  the  high  complexity  in  the  nature  of  government  acquisitions,  comparison  of 
proposals  and  the  source  selection  decision  involve  subjective  judgmental  evaluations, 
and  these  evaluations  may  be  in  conflict  with  fairness  and  the  best  value.4 

Considering  the  fact  that  resources  available  to  governments  are  becoming 
scarcer,  subjectivity  should  be  reduced  as  much  as  possible  in  contracting,  and  agencies 
should  find  some  ways  to  get  the  best  value  from  any  procurement. 


1  Carl  R.  Templin  and  Ken  R.  Noffsinger,  “An  Assessment  of  the  Role  of  Technical  and  Evaluation 
Factors,”  International  Journal  of  Purchasing  and  Materials  Management  30  (1994);  38. 

2  Federal  Acquisition  Regulation  (FAR)  15.305. 

3  John  Cibinic  and  Ralph  C.  Nash,  Formation  of  Government  Contracts,  3rd  ed.  (Washington,  DC: 
The  George  Washington  University,  1998),  821. 

4  Templin  and  Noffsinger,  “An  Assessment,”  38. 
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As  a  solution  to  the  problems  stated  above,  quantitative  methods  and  information 
technology  tools  can  be  utilized  to  augment  objectivity  in  the  source  selection  decision 
making.  Quantitative  methods  and  information  technology  tools  can  assist  with  decision¬ 
making  by  analyzing  data  and  providing  some  solutions.  Choosing  an  optimal  solution 
among  different  alternatives  using  many  evaluation  criteria,  performing  risk  assessments, 
and  planning  complicated  operations  are  some  advantages  of  information  technology 
tools.  “Implementation  and  use  of  these  tools  in  a  systematic  method  can  help  shift 
government  purchasing  from  a  tactical,  reactive  mode  of  operation  to  a  more  strategic, 
proactive  mode  of  operation.”5 

In  this  context,  the  aim  of  this  study  is  to  show  the  effects  of  the  subjectivity 
involved  in  the  government  procurement  processes  and  suggest  some  possible  solutions. 
The  authors  develop  and  analyze  multiple  versions  of  numerical  models  based  on  the 
weighted  sum  method,  which  is  one  of  the  tools  that  can  be  used  to  solve  multi-criteria 
decision  making  problems.  These  models  are  built  in  Microsoft  Excel  using  data  from  a 
current  procurement  program. 

In  this  study,  the  USAF’s  KC-X  Tanker  Replacement  Program  has  been  selected 
as  the  program  to  be  used  to  frame  the  research  questions.  There  are  some  specific 
reasons  for  choosing  that  program.  First  of  all,  an  Air  Force  procurement  program  in 
which  color  ratings  were  used  has  been  chosen  to  show  the  effects  of  subjectivity  in 
government  contracting  because,  in  the  authors’  perspective,  the  Air  Force’s  color  rating 
method  is  one  of  the  methods  that  is  incapable  of  considering  small  differences  between 
proposals  and  thus  causes  subjectivity  to  play  a  role  during  the  decision  process.  In 
addition,  a  popular  and  a  current  program  like  the  USAF’s  KC-X  program  has  been 
selected  to  be  able  to  reach  lots  of  different  comments  from  many  different  viewpoints. 
Considering  the  confidentiality  issues  of  past  procurement  programs  that  make  it 
impossible  to  reach  enough  information,  a  current  program  provides  much  more  publicly 
available  infonnation.  After  all  these  considerations,  the  authors  came  up  with  the  idea  of 
doing  their  study  on  the  USAF’s  KC-X  program. 

5  Kathy  L.  Spainhower,  “An  Exploratory  Study  on  the  Strategic  Use  of  Information  Technology,” 
(master’s  thesis.  Air  Force  Institute  of  Technology,  1998),  7. 
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After  doing  some  research  about  the  KC-X  program,  it  has  been  discovered  that 
the  two  different  proposals  offered  very  different  specifications  and  capabilities  for  the 
same  program.  This  showed  the  difference  between  the  two  offerors’  viewpoints  and 
made  it  clear  that  this  program  might  be  a  good  sample  for  this  study  because  of  the 
potential  effects  of  subjectivity. 

Another  factor  that  makes  this  choice  even  more  interesting  is  the  high  likelihood 
of  the  contract  award  being  protested.  Experts  of  the  defense  industry  were  expecting  a 
protest  regardless  of  the  winner.  Since  the  proposed  KC-X  alternatives  have  very 
different  strengths  in  capabilities,  the  results  would  not  easily  satisfy  the  offeror  that  loses 
the  program  award.  As  expected,  after  the  authors  started  working  on  the  KC-X  program 
to  show  the  subjectivity  involved  in  source  selection  (mostly  because  of  the  color  rating 
technique)  and  its  effects  on  the  source  selection  decision,  the  program  was  awarded  to 
the  Northrop  Grumman-Airbus  EADS  team  and  Boeing  immediately  protested  the 
program  to  the  Government  Accountability  Office  (GAO). 

In  this  study,  detailed  explanations  about  government  contracting  and  decision¬ 
making  support  techniques  are  presented  first  to  provide  readers  with  some  background 
information.  After  that,  the  USAF’s  KC-X  Tanker  Replacement  Program  is  examined  in 
detail  to  demonstrate  its  special  features  and  high  importance.  Also,  this  part  provides  the 
color  rating  data  of  both  offerors  that  were  found  in  publicly  available  sources  and  used 
in  the  models.  After  these  explanatory  chapters,  the  model  construction  chapter  shows 
how  the  two  versions  of  the  model  were  built  with  some  simple  examples.  In  the  chapter 
that  discusses  the  analysis  and  results,  it  is  shown  with  both  versions  of  the  model  that  the 
final  source  selection  result  of  the  KC-X  Tanker  Replacement  Program  could  have  been 
different  if  the  USAF  had  used  the  weighted  sum  method  as  its  source  selection 
evaluation  method.  Then,  optimization  is  built  into  the  models  to  perform  sensitivity 
analysis  with  respect  to  choices  that  were  made  when  constructing  the  models.  In  the 
final  part,  the  conclusions  from  each  part  of  the  study  are  summarized  and 
recommendations  for  further  research  are  presented  to  the  readers. 
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II.  ACQUISITION  OVERVIEW 


A.  INTRODUCTION 

Government  acquisitions  are  to  be  conducted  in  compliance  with  many  laws  and 
regulations  that  require  contracting  agencies  to  ensure  that  all  contractors  are  treated  in  an 
equal  manner  and  that  the  government  acquires  the  supplies  or  services  for  a  fair  and 
reasonable  price.6 

In  awarding  U.S.  government  contracts,  The  Competition  in  Contract  Act  of  1984 
(CICA)  requires  ‘full  and  open  competition’  to  be  provided  by  contracting  officers. 
According  to  CICA,  “full  and  open  competition  means  that  all  responsible  sources  are 
pennitted  to  submit  sealed  bids  or  competitive  proposals  on  the  procurement.”7  The 
importance  of  full  and  open  competition  for  government  is  that  if  performed  ideally,  it 
results  in  quality  services  and  products  provided  to  the  government  at  desirable  costs.  It 
has  been  experienced  that,  on  average,  25%  of  the  total  cost  can  be  saved  by  using 
competitive  processes.8 

The  source  selection  phase  in  government  contracting  involves  multi-criteria 
decision  making  that  has  to  deal  with  many  critical  factors,  such  as  cost,  technical 
capabilities,  and  past  performance  of  contractors.  The  contracting  processes  used  in 
government  acquisitions  have  been  built  to  respond  to  various  government  procurement 
needs  by  evaluating  these  factors. 

Under  some  requirements,  contracting  officers  may  use  contracting  procedures 
that  provide  no  full  and  open  competition  and  the  justifications  for  awarding  a  contract 
without  competition  is  stated  in  Federal  Acquisition  Regulation  (FAR)  6.303.  Based  on 


6  Templin  and  Noffsinger,  “An  Assessment,”  38. 

7  William  Thybony,  Government  Contracting  Based  on  the  Acquisition  Regulation  (and  the 
Competition  in  Contracting  Act  of 1984)  (Thybony,  tnc.,  1985),  95. 

8  Ibid.,  99. 
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the  circumstances,  “such  procurements  may  be  made  on  either  sole  source  basis  or  with 
limited  competition.”9  According  to  FAR  6.302,  situations  where  full  and  open 
competition  is  not  required  are  as  follows: 

•  “Only  one  responsible  source  and  no  other  supplies  or  services  will  satisfy 
agency  requirements 

•  Unusual  and  compelling  urgency 

•  Industrial  mobilization;  engineering,  developmental,  or  research 
capability;  or  expert  services 

•  International  agreement 

•  Authorized  or  required  by  statute 

•  National  security 

•  Public  interest” 

This  chapter  is  mainly  divided  into  two  parts.  The  first  part  (Section  B)  provides 
general  background  information  about  government  contracting.  While  both  methods  of 
government  contracting,  which  are  sealed  bidding  and  contracting  by  negotiations,  are 
explained  during  this  part  of  the  study,  the  main  focus  of  this  paper  is  on  Contracting  by 
Negotiations  because  it  is  the  technique  more  commonly  used  in  government  contracting 
practices. 

The  second  part  (Section  C)  provides  information  about  Air  Force  source 
selection  procedures  because  the  authors  are  going  to  use  a  U.S.  Air  Force  weapon 
system  acquisition,  namely  the  KC-X  Tanker  Replacement  Program,  as  the  study  case 
and  example  to  demonstrate  the  subjectivity  involved  in  government  source  selection 
decisions. 

B.  CONTRACTING  METHODS 

Contracting  officers  are  free  to  choose  the  method  of  contracting,  but  they  are 
required  to  choose  the  best  responsive  method  that  meets  all  government  requirements.10 
Basically,  the  complexity  of  the  acquisition  and  the  need  for  discussion  determine  the 

9  Cibinic  and  Nash,  Formation  of  Government  Contracts,  280. 

10  Ibid.,  312. 
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methods  of  contract  awarding.  For  instance,  in  non-complex  acquisitions  which  do  not 
require  discussion,  a  contract  can  be  awarded  to  the  lowest  priced  offeror  who  has 
proposed  to  provide  at  least  minimum  technical  requirements.  On  the  other  hand, 
complex  acquisitions,  for  instance  big  weapon  systems  procurements,  lead  contracting 
officers  to  use  other  options  that  help  to  detennine  the  offeror  whose  proposal  would  give 
more  value  to  the  government. 1 1 

In  order  to  accomplish  full  and  open  competition,  the  following  competitive 
procedures  are  used  according  to  FAR  6. 102: 

•  Sealed  bids 

•  Competitive  proposals 

•  Combination  of  competitive  procedures 

•  Other  competitive  procedures 

In  this  study,  the  first  three  will  be  examined  while  stressing  the  attributes  of  only 
the  sealed  bidding  and  the  competitive  procedures. 

1.  Sealed  Bidding 

The  purpose  of  sealed  bidding  is  to  encourage  competition  while  avoiding  fraud 
or  favoritism.12  While  determining  the  procedures  that  will  be  used  for  a  contract, 
according  to  FAR  6.401,  contracting  officers  shall  solicit  sealed  bids  if: 

•  “Time  permits  the  solicitation,  submission,  and  evaluation  of  sealed  bids; 

•  The  award  will  be  made  on  the  basis  of  price  and  other  price-related 
factors; 

•  It  is  not  necessary  to  conduct  discussions  with  the  responding  offerors 
about  their  bids, 

•  There  is  a  reasonable  expectation  of  receiving  more  than  one  sealed  bid.” 


11  Spainhower,  “An  Exploratory  Study,”  14. 

12  Noel  W.  Keyes,  Government  Contracts  under  the  Federal  Acquisition  Regulation  (St.  Paul,  MN: 
West  Publishing  Co.,  1986),  153. 
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In  sealed  bidding,  price  and  price-related  factors  play  a  great  role  in  choosing  the 
offeror.  According  to  FAR  14.103-2,  “an  award  is  made  to  the  responsible  bidder  whose 
bid  is  responsive  to  the  terms  of  the  invitation  for  bids  and  is  most  advantageous  to  the 
government,  considering  only  price  and  the  price  related  factors  included  in  the 
invitation.” 

The  price-related  factors  in  FAR  14.201-8  include: 

•  “Foreseeable  costs  or  delays  to  the  government  resulting  from  such  factors 
as  differences  in  inspection,  locations  of  supplies,  and  transportation. 

•  Changes  made,  or  requested  by  the  bidder,  in  any  of  the  provisions  of  the 
invitation  for  bids. 

•  Advantages  or  disadvantages  to  the  government  that  might  result  from 
making  more  than  one  award. 

•  Federal,  state,  and  local  taxes. 

•  Origin  of  supplies,  and,  if  foreign,  the  application  of  the  Buy  American 
Act  or  any  other  prohibition  on  foreign  purchases.” 

The  evaluation  process  becomes  more  complex  when  there  are  more  factors  to  be 
considered.  Multiple  items,  indefinite  quantities,  extended  performance  periods,  or  price 
adjustment  provisions  make  this  evaluation  process  more  complex  and  hard  to  manage. 
Additionally,  decision  making  turns  to  its  most  difficult  form  when  price-related  factors 
are  included  in  the  evaluation  formula.13  In  solicitation,  the  contracting  agency  must 
provide  all  price-related  factors  and  evaluation  methods  in  a  clear  manner  so  that  all 
bidders  can  easily  understand  how  their  proposals  will  be  evaluated.14 

When  two  or  more  bids  are  equal  after  the  evaluation,  the  contracting  agency  shall 
award  the  contract  based  on  the  following  order  of  priority: 

•  “Small  business  concerns  that  are  also  labor  surplus  area  concerns, 

•  Other  small  business  concerns, 

•  Other  business  concerns  that  are  also  labor  surplus  area  concerns, 

•  Other  business  concerns.”15 

13  Cibinic  and  Nash,  Formation  of  Government  Contracts,  593. 

14  Ibid.,  614. 

15  Thybony,  Government  Contracting,  147. 
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If  there  are  still  equal  bids  after  going  through  these  priority  steps,  the  contract  is  awarded 
by  a  drawing  witnessed  by  bidding  contractors  if  possible.16 

2.  Contracting  by  Negotiation 

Negotiated  contracting  includes  discussions  with  contractors  and  it  usually  gives 
an  opportunity  to  contractors  to  revise  their  offers  before  the  contract  is  awarded. 
According  to  FAR  15.302,  “the  objective  of  source  selection  in  negotiated  contracting  is 
to  select  the  proposal  that  represents  the  best  value.”  Best  value  basically  means  the 
greatest  benefit  that  the  government  can  get  from  an  acquisition. 

In  order  to  get  more  value  through  flexibility  obtained  by  negotiations,  statutory 
changes  require  discussions  while  awarding  a  contract,  more  detailed  information  about 
factor  and  subfactors  in  solicitations,  limitations  of  competitive  range,  and  detailed 
debriefings  of  offerors.  Also,  CICA  orders  competitive  procedures  or  a  combination  of 
procedures,  whichever  are  suitable,  to  be  used.17 

In  FAR  15.002,  in  order  to  get  the  benefits  of  the  flexibility  of  negotiation,  types 
of  negotiation,  which  are  competitive  and  sole  source  acquisitions,  are  described  in  a  way 
that  aims  to  reduce  complexity  due  to  complicated  solicitations  that  involve  unnecessary 
large  numbers  of  evaluation  factors  and  subfactors.18 

Contracting  agencies  can  use  any  process  or  a  combination  of  processes  to  get  the 
best  value  from  procurement.  These  processes  basically  involve  “trade-off’  and  “lowest 
price  technically  acceptable  source  selection.”19 

The  more  detailed  background  information  about  contracting  by  negotiation 
provided  after  this  point  will  be  divided  into  three  sublevels,  which  are:  selection 
processes,  evaluation  factors  and  subfactors  and  their  relative  importance,  and  evaluation 
of  factors. 


16  Thybony,  Government  Contracting,  148. 

17  Ibid.,  313. 

18  Cibinic  and  Nash,  Formation  of  Government  Contracts,  710. 

19  FAR  15.101. 
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a.  Selection  Processes 

Before  soliciting  proposals,  contracting  agencies  shall  determine  the 
strategy  that  will  be  followed  to  make  the  source  selection  decision.  The  strategy  that  is 
used  by  agencies  is  also  called  “decisional  rule.”20 

A  solicitation,  at  least,  should  state  that  “evaluation  factors  other  than  cost 
or  price,  when  combined,  are- 

•  significantly  more  important  than  cost  or  price, 

•  approximately  equal  in  importance  to  cost  or  price;  or 

•  significantly  less  important  than  cost  or  price.”21 

The  expected  outcome  of  any  strategy  is  to  give  the  government  the  best 
value  from  any  procurement.  Best  value  is  the  greatest  benefit  that  can  be  obtained  in 
response  to  the  requirement. 

In  FAR  15.101,  “best  value”  is  explained  as  follows: 

An  agency  can  obtain  best  value  in  negotiated  acquisitions  by  using  any 
one  or  a  combination  of  source  selection  approaches.  In  different  types  of 
acquisitions,  the  relative  importance  of  cost  or  price  may  vary.  For 
example,  in  acquisitions  where  the  requirement  is  clearly  definable  and  the 
risk  of  unsuccessful  contract  performance  is  minimal,  cost  or  price  may 
play  a  dominant  role  in  source  selection.  The  less  definitive  the 
requirement,  the  more  development  work  required,  or  the  greater  the 
performance  risk,  the  more  technical  or  past  perfonnance  considerations 
may  play  a  dominant  role  in  source  selection. 

(1)  Trade-off  Process.  Contracting  by  negotiation  allows 
contracting  agencies  to  make  trade-offs  between  cost  and  other  factors."  This  process 
aims  to  find  the  best  value  proposal  by  comparing  the  differences  between  technical 
specifications  and  costs  and  to  justify  paying  more  than  the  lowest  price.  Therefore, 


20  Cibinic  and  Nash,  Formation  of  Government  Contracts,  712. 

21  Ibid. 

22  Ibid.,  710. 
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contracting  agencies  may  not  always  select  the  lowest-priced  bidder.  A  contract  may  be 
awarded  to  a  higher-priced  bidder  if  the  justification  is  that  of  better  quality  of  the 

23 

proposed  product  or  serviced 

This  is  explained  in  FAR  15.101-1  as  follows: 

(a)  “A  tradeoff  process  is  appropriate  when  it  may  be  in  the  best 
interest  of  the  Government  to  consider  award  to  other  than 
the  lowest  priced  offeror  or  other  than  the  highest  technically 
rated  offeror. 

(c)  This  process  permits  tradeoffs  among  cost  or  price  and  non¬ 
cost  factors  and  allows  the  Government  to  accept  other  than 
the  lowest  priced  proposal.  The  perceived  benefits  of  the 
higher  priced  proposal  shall  merit  the  additional  cost,  and  the 
rationale  for  tradeoffs  must  be  documented.” 

(2)  Lowest  Price  Technically  Acceptable  Source  Selection  Process. 
In  this  process,  the  best  value  concept  is  accomplished  when  the  strategy  is  chosen,  not 
after  or  during  evaluation  of  the  proposals.24  In  other  words,  it  is  certain  that  the  offeror 
that  proposes  a  technically  acceptable  product  or  service  with  the  lowest  price  gives  the 
possible  best  value  to  the  government  in  this  process.  FAR  15.101-2  describes  the 
process  as  follows: 

(a)  The  lowest  price  technically  acceptable  source  selection  process  is 
appropriate  when  best  value  is  expected  to  result  from  selection  of  the 
technically  acceptable  proposal  with  the  lowest  evaluated  price. 

When  evaluating  proposals  that  meet  or  exceed  requirements  for 
non-cost  factors  based  on  price,  non-cost  factors  have  equal  importance.  Therefore,  if  a 
proposal  does  not  meet  any  of  the  non-cost  factors,  the  offeror  with  that  deficient 
proposed  product  or  service  loses  the  contract.25 


22  John  R.  Tmmm,  A  Decision  Analysis  Tool  for  the  Source  Selection  Process  (Air  Force  Institute  of 
Technology,  2006),  18. 

24  Cibinic  and  Nash,  Formation  of  Government  Contracts,  715. 

25  Ibid.,  716. 
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(3)  Combination  of  Trade-off  and  Low  Price  Acceptable 
Processes.  Although  FAR  does  not  provide  any  explanation  in  regards  to  combined 
processes,  in  part  15.101  it  allows  them  to  be  used.  In  this  process,  while  some  factors 
are  evaluated  based  on  a  “go,  no-go”  basis,  others  are  evaluated  on  their  relative  values. 
Otherwise,  as  another  strategy,  first  all  factors  are  evaluated  based  on  a  “go,  no-go”  basis 
and  the  proposals  that  can  pass  this  first  step  are  evaluated  on  their  relative  merit.26 

b.  Evaluation  Factors  and  Subfactors  and  Their  Relative 

Importance 

Evaluation  factors  and  subfactors  are  milestones  in  a  contract  that  describe 
what  is  significant  for  the  procurement  needs.  These  factors  and  their  relative  importance 
should  be  clearly  described  in  solicitation  to  be  completely  understood  by  offerors. 

According  to  FAR  15.304,  “these  factors  and  subfactors  must 

•  represent  the  key  areas  of  importance  and  emphasis  to  be 
considered  in  the  source  selection  decision;  and 

•  support  meaningful  comparison  and  discrimination  between  and 
among  competing  proposals.” 

According  to  FAR  15.304,  contracting  officers  are  free  to  choose  these 
factors  and  determine  their  relative  importance.  However,  there  are  some  requirements  to 
be  followed  related  to  these  factors: 

•  In  all  government  contracts,  cost  should  be  evaluated. 

•  One  or  more  non-cost  factors  should  be  evaluated  to  ensure  the 
quality  of  products  or  services. 

•  Past  performance  shall  be  evaluated  unless  the  contracting  officer 
documents  that  it  is  not  an  appropriate  evaluation  factor. 

The  relative  importance  of  the  evaluation  factors  should  be  stated  in  two 
steps  in  the  request  for  proposal  (RFP).  In  the  first  step,  the  relationship  of  cost/price  to 
the  non-price  factors  should  be  pointed  out  and  then,  as  a  second  step,  the  relative 
importance  of  the  factors  and  subfactors  in  the  non-cost  or  price  areas  should  be 
described. 

26  Cibinic  and  Nash,  Formation  of  Government  Contracts,  718. 
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(1)  Relationship  of  Cost/Price  to  Other  Factors.  An  RFP  should,  at 
least,  involve  a  statement  indicating  the  decisional  logic  that  will  be  used  to  make  the 
source  selection  decision.  It  is  essential  for  establishing  competition  to  inform  each 
offeror  whether  the  award  will  be  made  based  on  the  lowest-price  acceptable  proposal 
basis  or  trade-off  basis.  27 

Variable  relationships  and  fixed  relationships  can  be  used  to 
determine  the  importance  of  the  factors.  A  variable  relationship  is  usually  preferred  when 
the  cost  to  government  will  vary  depending  on  the  other  factors.  Also,  variable 
relationships  among  other  factors  are  possible.  28 

(2)  Relative  Importance  of  Other  Factors.  There  are  a  lot  of 
different  methods  to  determine  the  relative  importance  of  factors  other  than  cost  or  price. 

One  of  the  simplest  ways  to  disclose  the  relative  importance  of 
factors  is  to  list  the  factors  that  will  be  used  in  the  evaluation.  With  this  method,  no 
explanation  of  differences  between  factors  is  provided  and  the  offerors  may  assume  that 
all  evaluation  factors  have  approximately  equal  importance  in  determining  the  winning 
proposal  29 

Another  way  to  indicate  the  relative  importance  is  to  list  the  factors 
in  descending  order  based  on  their  importance  without  assigning  a  weight  to  each  of 
them.  This  method  is  useful  when  there  is  “a  reasonable  downward  progression  of 
evaluation  weights.”30  On  the  other  hand,  this  method  is  satisfactory  when  the  difference 
in  the  importance  of  two  factors  following  one  another  is  small.  Besides,  if  a  factor  has 
more  importance  compared  to  others,  this  should  be  disclosed  in  the  RFP.31 


27  Cibinic  and  Nash,  Formation  of  Government  Contracts,  748. 

28  Ibid. 


29  Ibid.,  751. 

30  Ibid. 

31  Ibid. 
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c.  Evaluation  of  Factors  and  Subfactors 

According  to  FAR  15.305,  “evaluations  may  be  conducted  using  any 
rating  method  or  combination  of  methods,  including  color  or  adjectival  ratings,  numerical 
weights,  and  ordinal  rankings.” 

Additionally,  DoD  Directive  4105.62  states: 

There  is  no  prescribed  methodology  for  rating.  Past  practices  include  color 
coding,  numerical,  and  plus  or  minus  checks.  The  important  thing  is  not 
the  rating  methodology  but  the  consistency  with  which  it  is  applied  to 
elements  of  proposals  and  among  proposals  to  ensure  a  though  and  fair 
evaluation.  Evaluators  must  be  well  rounded  in  their  field  of  expertise  and 
be  able  to  apply  mature  professional  judgment.... evaluators  must  support 
the  rating  assigned  with  a  concise  narrative  that  addresses  strengths, 
weaknesses,  and  risks  in  the  proposal.32 

As  advised  by  these  regulations,  contracting  agencies  can  use  words  or 
symbols  to  describe  the  compatibility  levels  of  each  evaluation  factor  and  this  method  is 
called  adjectival  ranking.  For  instance,  the  Air  Force  uses  a  descriptive  adjective  system, 
a  color  coding  system,  and  a  symbol  system.33  Since  the  Air  Force  source  selection  rating 
system  is  one  the  most  significant  cornerstones  in  this  study,  more  focus  will  be  given  to 
it  and  explanations  about  it  will  be  provided  throughout  the  study. 

Another  method  for  evaluating  the  factors  is  to  use  numeric  point  scores  or 
percentages  for  each  of  them.  This  method  may  be  a  good  guide  for  source  selection 
authorities  while  selecting  the  best  offeror,  but  it  is  warned  that  it  should  not  be  used  as  a 
primary  tool  in  the  decision-making  process.  It  is  stated  that: 

Numerical  point  scores,  when  used  for  proposal  evaluation,  are  useful  as 
guides  to  intelligent  decision  making,  but  are  not  themselves  controlling  in 
determining  award,  since  these  scores  can  only  reflect  the  disparate, 
subjective,  and  objective  judgments  of  the  evaluators.  Whether  a  given 
point  spread  between  competing  offers  indicates  the  significant  superiority 
of  one  proposal  over  another  depends  on  the  facts  and  circumstances  of 


32  Spainhower,  “An  Exploratory  Study,”  22. 

33  Ibid. 
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each  procurement,  and  while  technical  scores  must  of  course  be 
considered  by  Source  Selection  Authorities  (SSAs),  such  officials  are  not 
bound  thereby.34 

Table  1  below  shows  a  combination  of  adjectival  and  numerical 


systems:35 


Numerical 

Scores 

Adjective 

Rating 

Definition  /  Evaluation 

10 

Excellent 

Innovative,  comprehensive  and  complete  in  all  details, 

meets  all  requirements  and  objectives  without  “gold 

plating” 

9 

Very  Good 

Substantial  response  in  clearly  definable  detail,  meets 

all  critical  requirements 

7 

Average 

Generally  meets  minimum  requirements 

6 

Poor 

Lack  of  essential  infonnation  to  substantiate  data 

presented 

5 

Unsatisfactory 

Lack  of  understanding  of  requirements  or  omissions  in 

major  areas 

0 

No  data 

Table  1.  Sample  Cross  Reference  of  Numerical  and  Adjectival  Scoring  Systems. 


C.  SOURCE  SELECTION  IN  UNITED  STATES  AIR  FORCE 

The  objective  of  source  selection  in  the  U.S.  Air  Force  is  to  select  the  offeror 
which  proposes  the  best  value  to  the  government.  In  major  weapon  system  procurements, 
the  Air  Force  uses  contracting  by  negotiation  procedures  to  encourage  discussions  and 
negotiations  to  reach  the  best  value.  The  confidence  in  an  offeror’s  ability  to  meet  all 

34  Spainhower,  “An  Exploratory  Study,”  23. 

35  Ibid. 
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requirements  perfectly  is  the  key  in  a  best  value  continuum,  and  this  confidence  may  lead 
SSA  to  award  a  contract  to  an  offeror  which  is  not  the  one  offering  the  lowest  cost  to  the 
government.  On  the  other  hand,  openness  and  early  industry  participation  are  the  main 
points  associated  with  the  relations  with  industry  in  the  acquisitions  of  the  Air  Force.36 

1.  Categories  of  Air  Force  Acquisitions 

There  are  three  categories  of  Air  Force  source  selection  based  on  the  dollar  value 
and  complexity  of  the  acquisition.  The  categories  are  as  follows: 

•  Basic  source  selection 

•  Median  source  selection 

•  Agency  source  selection 

Basic  source  selection  is  the  easiest  one  among  all  three  types  and  the  associated 
procedures  are  simple,  direct  and  minimal.  For  non-information  technology  negotiated 
acquisitions,  the  threshold  is  $10  million  and  for  information  technology  efforts,  the  cost 
range  should  be  less  than  $15  million  per  one  year  and  less  than  $30  million  for  the 
whole  program.37 

Median  source  selection  is  used  for  moderately  complex  source  selections  and 
needs  a  more  structured  approach  than  basic  source  selections.  Cost/price,  past 
performance,  mission  capability,  and  proposal  risk  are  always  used  as  factors  in  this  type 
of  source  selection.  For  non-information  technology  negotiated  acquisitions,  the 
minimum  threshold  is  $10  million  and  the  maximum  threshold  is  $100  million.  On  the 
other  hand,  for  information  technology  efforts,  “these  procedures  apply  to  Non-Major 
Automated  Information  Systems  (MAIS)  that  are  equal  to  or  greater  than  $15  million  in  a 
fiscal  year  or  are  equal  to  or  greater  than  $30  million  for  the  total  program  but  less  than 
$120  million.”38 


36  Air  Force  Source  Selection  Procedures  Guide,  3,  https://www.afinc-mil.wpafb.af.mil/HQ- 
AFMC/PK/pkp/polvault/guides/sspguide.doc  (accessed  December  20,  2007). 

37  Ibid.,  5. 

38  Ibid.,  6. 
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Agency  source  selection  procedures  are  used  for  the  most  complex  acquisitions 
that  need  a  more  structured  approach  than  median  source  selections.  Under  the  mission 
capability  factor,  there  are  several  subfactors  and  elements  to  be  evaluated  if  necessary. 
For  non- information  technology  negotiated  acquisitions,  the  minimum  threshold  is  $100 
million.  For  information  technology  acquisitions,  the  minimum  threshold  is  $120  million 
to  use  agency  source  selection  procedures.39 

2.  Steps  in  Air  Force  Source  Selection 

All  these  three  types  of  source  selection  procedures  are  composed  of  the  same 
three  steps,  which  are: 

1 .  Pre-Solicitation  activities 

2.  Evaluation  activities 

3.  Award  activities 

In  this  study,  because  the  authors  are  going  to  use  a  major  weapon  system 
acquisition  as  an  example,  the  cost  of  which  exceeds  billions  of  dollars,  the  agency 
source  selection  procedure  will  be  the  focus  of  discussion. 

a.  Pre-Solicitation  Activities 

After  detennining  the  requirements,  early  participation  of  the  industry  and 
use  of  oral  presentations  instead  of  written  statements  are  recommended  in  this  stage  of 
source  selection.  The  other  significant  activity  is  to  identify  high-risk  areas  and  determine 
discriminators  for  the  selection.  Detennining  these  important  factors  makes  it  possible  to 
build  the  basis  for  award,  evaluation  criteria,  and  evaluation  factors  which  are  announced 
in  the  source  selection  plan. 

(1)  Evaluation  Factors.  The  following  four  factors  are  used  in  Air 
Force  source  selection  procedures: 

•  Mission  capability 

•  Proposal  risk 


39  Air  Force  Source  Selection  Procedures  Guide,  7. 
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•  Past  performance 

•  Price  or  cost  evaluation  factor 

The  mission  capability  factor  focuses  on  the  technical  requirements 
that  have  significant  value  in  the  source  selection  decision.  If  subfactors  under  this  factor 
are  to  be  used,  the  number  of  subfactors  can  not  exceed  six.40  The  need  for  using 
subfactors  and  elements  are  determined  by  performing  a  formal  risk  assessment.  Keeping 
the  numbers  of  subfactors  and  elements  low  helps  the  evaluation  team  make  a  meaningful 
and  complete  evaluation.  41 

The  proposal  risk  factor  focuses  on  the  risks  an  offeror  puts  on  the 
program’s  significant  goals,  which  are  cost,  schedule  and  performance.  This  factor  is  not 
mandatory  for  acquisitions  of  $10  million  or  below.42 

The  past  performance  factor  focuses  on  the  confidence  that  an 
offeror  provides  to  the  government  showing  the  offeror’s  ability  to  supply  products  or 
services  while  meeting  all  requirements  stated  in  the  solicitation.  It  may  be  the  most 
important  factor  or  at  least  as  important  as  the  most  important  non-cost  factor  in  the 
evaluation  43  No  subfactor  is  typically  assigned  to  this  factor  like  the  price/cost  factor.44 

Price/cost  evaluation  focuses  only  on  reasonableness  and  realism 
aspects  of  the  price/cost  proposed  by  an  offeror.  Therefore,  the  data  requested  from  an 
offeror  to  perform  this  evaluation  should  be  limited  to  the  amount  required  to  perform  the 
reasonableness  and  realism  assessment.45 


40  Air  Force  Mandatory  Source  Selection  Procedures  4.4. 1.1, 
http://farsite.hill.af.mil/reghtmFregs/far2afmcfars/af  afmc/affars/5315.htm#P41  1723  (accessed  January 
14,  2007). 

41  Air  Force  Source  Selection  Procedures  Guide,  43. 

42  Air  Force  Mandatory  Source  Selection  Procedures  4.4. 1.2. 

43  Air  Force  Source  Selection  Procedures  Guide,  42. 

44  Alexander  R.  Slate,  “Best  Value  Source  Selection:  The  Air  Force  Approach,  Part  1,”  Defense  AT&L 
(September-October  2004);  52. 

45  Air  Force  Source  Selection  Procedures  Guide,  43. 
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After  determining  all  factors,  subfactors,  and  elements,  as  well  as 
their  relative  importance,  a  formal  request  for  proposal  is  issued  and  released. 

b.  Evaluation  Factor  Assessment  Activities 

The  goal  of  source  selection  evaluation  in  Air  Force  acquisitions  is  to 
provide  the  best  value  to  the  government.  Proposals  given  by  offerors  can  only  be 
evaluated  based  on  the  factors,  subfactors  and  elements,  if  applicable,  included  in  the 
RFP. 


(1)  Mission  Capability  Assessment.  The  mission  capability 
assessment  focuses  on  the  deficiencies  and  strengths  in  the  offeror’s  proposal.  If  there  are 
subfactors  established  under  this  factor,  they  should  be  rated  at  a  subfactor  level  and  an 
overall  factor  rating  is  not  assigned.  This  factor  is  rated  using  colors  and  each  of  the 
mission  capability  subfactors  should  get  a  color  coding  based  on  the  performance 
capability  an  offeror  proposes.46  Table  2  depicts  the  color  ratings  and  their  descriptions 
used  in  this  evaluation. 


MISSION  CAPABILITY  COLOR  RATINGS 

Color 

Rating 

Description 

Blue 

Exceptional 

Exceeds  specified  minimum  performance  or  capability 
requirements  in  a  way  beneficial  to  the  government; 
proposal  must  have  one  or  more  strengths  and  no 
deficiencies  to  receive  a  blue  rating. 

Green 

Acceptable 

Meets  specified  minimum  performance  or  capability 
requirements  delineated  in  the  RFP;  proposal  rated  green 
must  have  no  deficiencies  but  may  have  one  or  more 
strengths. 

Yellow 

Marginal 

Does  not  clearly  meet  some  specified  minimum  performance 
or  capability  requirements  delineated  in  the  RFP,  but  any 
such  uncertainty  is  correctable. 

46  Air  Force  Mandatory  Source  Selection  Procedures  5.5.1. 
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Red 


Unacceptable 


Fails  to  meet  specified  minimum  performance  or  capability 
requirements;  proposal  has  one  or  more  deficiencies. 
Proposals  with  an  unacceptable  rating  are  not  awardable. 


Table  2.  Mission  Capability  Color  Ratings. 


A  yellow/marginal  rating  should  be  eliminated  through 
information  exchanges  and  at  the  end  of  the  final  evaluation,  it  should  be  rare.47  If  a  red 
rating  remains,  that  proposal  can  not  be  awarded  the  contract. 

Along  with  color  ratings,  SSA  should  be  informed  with  narrative 
assessments,  and  those  assessments  should  not  compare  offerors,  they  should  only 
evaluate  each  offeror’s  proposal  against  requirements  established  in  the  RFP.48 

(2)  Proposal  Risk  Assessment.  Proposal  risk  assessment  focuses 


on  the  weaknesses  in  the  proposed  approach.  It  should  be  evaluated  and  rated  at  the 
subfactor  level  against  the  mission  capability  subfactors,  and  an  overall  rating  is  not 
assigned.49  Table  3  demonstrates  the  ratings  used  in  this  evaluation. 


PROPOSAL  RISK  RATINGS 

Rating 

Description 

High 

Likely  to  cause  significant  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Risk  may  be  unacceptable  even  with  special  contractor  emphasis  and 
close  government  monitoring. 

Moderate 

Can  potentially  cause  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Special  contractor  emphasis  and  close  government  monitoring  will  likely 
be  able  to  overcome  any  difficulties. 

Low 

Has  little  potential  to  cause  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Normal  contractor  effort  and  normal  government  monitoring  will  likely  be 
able  to  overcome  any  difficulties. 

Table  3.  Proposal  Risk  Ratings. 


47  Air  Force  Mandatory  Source  Selection  Procedures  5.5.1. 

48  Air  Force  Source  Selection  Procedures  Guide,  46. 

47  Air  Force  Mandatory  Source  Selection  Procedures  5.5.2. 
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(3)  Past  Performance  Assessment.  Past  performance  evaluation 
demonstrates  the  government’s  confidence  in  an  offeror’s  ability  to  meet  all  requirements 
established  in  the  RFP.  The  most  critical  aspects  of  the  information  obtained  about  the 
offeror’s  past  performance  are  recency  and  relevancy.  In  agency  source  selection,  past 
performance  should  be  evaluated  by  a  Perfonnance  Risk  Assessment  Group  (PRAG).^ 

A  confidence  rating  is  assigned  to  each  offeror  to  assess  recent  past 
performance,  with  focus  on  the  performance  related  to  the  mission  capability  subfactors  and 

cost  or  price. ^  1  The  six  confidence  assessment  ratings  are  seen  in  Table  4. 


PERFORMANCE  CONFIDENCE  ASSESSMENTS 

Rating 

Description 

High  Confidence 

Based  on  the  offeror’s  performance  record,  the  government  has  high 
confidence  the  offeror  will  successfully  perform  with  the  required  effort. 

Significant  Confidence 

Based  on  the  offeror’s  performance  record,  the  government  has  significant 
confidence  the  offeror  will  successfully  perform  with  the  required  effort. 

Satisfactory  Confidence 

Based  on  the  offeror’s  performance  record,  the  government  has  confidence 
the  offeror  will  successfully  perform  the  required  effort.  Normal  contractor 
emphasis  should  preclude  any  problems. 

Unknown  Confidence 

No  performance  record  is  identifiable. 

Little  Confidence 

Based  on  the  offeror’s  performance  record,  substantial  doubt  exists  that  the 
offeror  will  successfully  perform  the  required  effort. 

No  Confidence 

Based  on  the  offeror’s  performance  record,  extreme  doubt  exists  that  the 
offeror  will  successfully  perform  the  required  effort. 

Table  4.  Performance  Confidence  Assessments. 

(4)  Cost/Price  Assessment.  Cost/price  assessment  focuses  on  the 


fairness  and  realism  of  the  cost/price  data  provided.  A  risk  rating  should  be  assigned  to 
this  factor  to  assess  the  risk  involved  to  the  proposed  cost.  Table  5  depicts  the  ratings 
used  in  the  cost/price  risk  assessment. 


50  Air  Force  Source  Selection  Procedures  Guide,  45. 

Air  Force  Mandatory  Source  Selection  Procedures  5.5.3. 
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COST/PRICE  RISK  RATINGS 

Rating 

Description 

High 

Likely  to  cause  significant  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Risk  may  be  unacceptable  even  with  special  contractor  emphasis  and  close 
government  monitoring. 

Moderate 

Can  potentially  cause  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Special  contractor  emphasis  and  close  government  monitoring  will  likely 
be  able  to  overcome  any  difficulties. 

Low 

Has  little  potential  to  cause  disruption  of  schedule,  increased  cost,  or  degradation  of 
performance.  Normal  contractor  effort  and  normal  government  monitoring  will  likely  be 
able  to  overcome  any  difficulties. 

Table  5.  Cost/Price  Risk  Ratings. 

Before  awarding  the  contract,  it  may  be  necessary  to  exchange 


some  information  with  offerors,  and  these  information  exchanges  are  composed  of 
“clarifications”,  “communications”,  and  “discussions.”  After  finishing  these  activities, 
final  proposals  are  evaluated  by  the  evaluation  team. 

c.  Award  Activities 

After  proposals  are  received,  an  evaluation  team  that  consists  of  many 
experts  performs  a  very  detailed  assessment  and  reaches  the  final  color/adjectival  ratings 
and  narrative  assessments  for  each  offer.  Before  SSA  makes  the  source  selection 
decision,  the  evaluation  team  gives  him  a  briefing  containing  all  these  assessments.  This 
briefing  includes  the  strengths,  deficiencies,  and  weaknesses  an  offeror  has  in  its  proposal 

as  the  narrative  assessments.  The  SSA  evaluates  all  color/adjectival  ratings  and  narrative 
assessments  provided  by  the  evaluation  team  and  awards  the  contract  to  the  offeror  that 
proposes  the  best  value  to  the  government.52 

After  announcing  the  results  and  the  winner,  a  debriefing  that  provides  all 
evaluations  with  required  explanations  is  presented  to  each  offeror.  5  3 

52  Air  Force  Source  Selection  Procedures  Guide,  49. 

53  Ibid.,  50. 


22 


D. 


SUMMARY 


In  this  chapter,  a  general  overview  of  acquisition  processes  is  provided.  First, 
contracting  methods  are  discussed  and  then  evaluation  and  selection  processes  are 
defined  briefly.  In  the  second  part  of  this  chapter,  because  this  project  will  focus  on  Air 
Force’s  color  rating  method,  Air  Force  source  selection  is  examined  in  greater  detail  in 
tenns  of  categories  of  Air  Force  acquisitions,  steps  in  Air  Force  source  selection, 
evaluation  activities,  and  award  activities.  The  next  chapter  will  discuss  the  decision¬ 
making  support  techniques  that  allow  a  final  assessment  to  be  reached  from  the 
individual  assessments  explained  in  this  chapter. 
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III.  DECISION  MAKING  SUPPORT  TECHNIQUES  OVERVIEW 


A.  INTRODUCTION 

Decision  makers  are  trying  to  make  more  accurate  decisions  and  their  only  assets 
in  doing  so  are  their  personal  experience  and  the  available  information  about  the 
situation.  For  accurate  decisions,  decision  makers  need  relevant  and  correct  information, 
and  they  need  to  analyze  it  correctly  to  come  up  with  a  beneficial  final  decision.  With  the 
help  of  information  technology,  which  brings  a  lot  of  support  techniques  to  help  them 
make  better  analysis,  decision  makers  can  make  better  decisions. 

In  this  part  of  the  authors’  study,  they  will  give  some  background  information 
about  the  available  techniques  to  solve  a  multi-criteria  problem  because  the  example 
study  case  (the  Air  Force’s  KC-X  program)  in  this  thesis  is  a  major  weapon  system 
procurement  and  has  more  than  one  criterion  to  consider  while  trying  to  come  up  with  the 
best  value  product  for  the  government.  In  general,  decision  makers  have  two  different 
choices  to  use  to  solve  multi-criteria  problems.  A  decision  maker  can  decide  on  using  a 
multi-objective  optimization  or  a  multi-criteria  decision  making  technique.  Under  multi¬ 
criteria  decision  making,  there  are  many  different  techniques  available,  but  the  authors 
choose  the  most  popular  eight  methods  to  provide  background  information  about.  Also, 
there  are  many  decision  support  software  programs  in  the  literature.  These  software 
programs  are  supporting  tools  that  use  one  or  some  of  the  techniques,  but  the  decision 
maker  does  not  need  to  know  anything  about  the  techniques  to  use  these  programs.  The 
authors  will  talk  about  two  of  them  that  best  fit  their  needs.  The  authors  will  give  a  brief 
summary  of  each  technique  and  as  a  conclusion  they  will  make  the  detennination  about 
what  technique  they  will  use  in  their  study.  However,  the  aim  of  this  chapter  is  not  to 
provide  guidance  about  choosing  the  appropriate  technique  where  it  supports  the  most 
accurate  results  for  that  special  condition.  Also,  in  the  literature  of  techniques,  it  was 
stated  that  there  are  different  conditions  which  make  one  of  the  techniques  the  most 
appropriate  for  that  condition.  Most  of  the  literature  studies  do  not  provide  the  conditions 
that  make  the  techniques  the  most  appropriate  ones  for  that  specific  condition. 
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B. 


OVERVIEW 


There  are  some  different  alternative  techniques  to  help  and  support  decision 
makers  when  they  need  to  make  a  decision  in  real-life  situations.  Real-life  situations 
consist  of  multiple  criteria  which  often  conflict  with  each  other.  In  a  multiple  criteria 
decision  making  environment,  there  are  two  main  options  of  techniques  to  choose  from 
for  a  decision  maker.  A  decision  maker  can  choose  any  of  the  techniques  under  the  multi¬ 
criteria  decision-making  umbrella  or  any  of  the  optimization  techniques,  or  otherwise  any 
of  the  software  under  the  decision  support  system  or  expert  systems  umbrella  can  be 
chosen  to  support  the  decision  whenever  a  decision  maker  does  not  want  to  use  any  of  the 
techniques. 

The  chart  below  in  Figure  1  is  the  summary  of  options  under  a  hierarchical  order 
for  a  decision  maker  to  support  his  or  her  decisions  for  multi-criteria  situations. 


MULTI-CRITERIA 
DECISION  MAKING 


Multi-Objective  Multi-Criteria 

Optimization  Decision  Making 

Methods 


WSM  AHP  Outranking  TOPSIS  CP  MAUT 

or  Methods 

WPM 


PROMETHEE  ELECTRE 


Figure  1 .  Hierarchical  Order  for  Multi-criteria  Decision  Making. 
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1. 


Multi-Objective  Optimization 


The  basic  constrained  single  objective  optimization  problem  is  formulated  as 
follows: 


Min 

J=fW 

s.t. 

h{X )  =  0 

g(X)<  0 

D.V. 

xL<x<xv 

where  J  is  the  objective  function  and  J  equals  to  a  real  valued  function /(Aj),  X  is  a  vector 
of  n  real  independent  variables  called  decision  variables,  h(X)  and  g(X)  are  representing 
the  feasible  set  for  X  ( h  represents  the  set  of  equality  constraint  functions  and  g  represents 
the  set  of  inequality  constraint  functions  of  X),  and  Xi  <=X<=Xu  represents  the  lower  and 
upper  simple  boundaries  on  feasible  values  of  X. 

There  are  several  different  methods  which  can  be  used  for  solving  optimization 
problems  like  the  simplex  method  for  linear  constrained  problems,  sequential  quadratic 
programming  (SQP)  for  nonlinear  constrained  problems,  and  branch-and-bound 
techniques  for  discrete  variables  depending  on  the  types  of  objectives,  constraints  and 
variables.54 

However,  real-world  applications  often  imply  multiple  objectives.  For  example,  in 
product-design  optimization,  the  cost  and  the  quality  of  products  are  two  conflicting 
objectives.55  Another  example  from  real  engineering  design  decisions  typically  involve 
both  technical  (i.e.,  maximize  performance)  and  economic  (i.e.,  minimize  cost) 
considerations  at  the  same  time.  In  general,  performance  and  cost  are  also  opposing 
objectives,  like  quality  and  cost,  and  performance  can  only  be  increased  by  increasing 
cost.  In  these  cases,  the  optimization  problem  becomes  multi-objective.56 

54  Robert  A.  Wolf,  “Multi-Objective  Collaborative  Optimization  of  Systems  of  Systems,”  (master’s 
thesis,  Massachusetts  Institute  of  Technology,  2005),  24. 

55Dirk  Buche,  Peter  Stoll,  and  Petros  Koumoutsakos,  “An  Evolutionary  Algorithm  for  Multi-Objective 
Optimization  of  Combustion  Processes,”  Center  for  Turbulence  Research  Annual  Research  Briefs  (2001); 
231,  http://ctr.stanford.edu/ResBriefs01/bueche.pdf  (accessed  December  25,  2007). 

56  Wolf,  “Multi-Objective  Collaborative,”  24. 
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The  typical  multi-objective  formulation  is  given  as  follows: 


Min 

s.t. 

D.V. 


h{X)  =  0 
g(X)<0 

xL<x<xv 


where  J  and  equally  f(X)  is  a  function  (vector)  of  n  different  objective  functions  at 
solution  X.  The  solution  to  the  multi-objective  optimization  problem  is  a  set  of  Pareto 
points.  Pareto  solutions  are  those  for  which  improvement  in  one  objective  can  only  occur 
with  the  worsening  of  at  least  one  other  objective.  Thus,  instead  of  a  unique  solution  to 
the  problem  as  in  a  single  objective  optimization,  the  solution  to  a  multi-objective 
problem  is  a  set  of  Pareto  points.  While  working  on  multiple  objectives,  generally  a 
single  solution  that  optimizes  all  objectives  simultaneously  does  not  exist.  There  are 
multiple  solutions  that  are  "optimal"  with  respect  to  the  different  objective  functions. 
Generally,  the  best  solutions  are  the  ones  that  represent  a  compromise  between  the 
various  objectives.57 

As  a  multi-objective  problem  creates  multiple  solutions,  in  order  to  come  up  with 
a  single  preferred  solution,  some  method  must  be  chosen  to  decide  the  best  compromise 
which  maximizes  the  utility  function  below.  With  this  equation,  multiple  objective  values 
can  be  combined  into  a  single  utility. 

«[/(x)]=«[/1(x),/2(x),...,/„(x)] 


The  methods  to  reach  a  single  preferred  solution  can  be  divided  into  four  primary 
classes:58 

•  No  preference  methods  (the  distance  to  the  ideal  point  is  minimized-no 
input  from  decision  maker) 

•  A  priori  methods  (decision  makers  define  the  utility  function  before  any 
analysis) 


57  Wolf,  “Multi-Objective  Collaborative,”  25. 

58  Ibid. 
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•  Iterative  methods  (identify  decision  maker  preferences  progressively  as 
the  design  analysis  progresses) 

•  Posterior  methods  (identify  the  Pareto-optimal  frontier  and  then  allow  the 
decision  maker  to  determine  the  best  compromise  solution) 

2.  Multi-Criteria  Decision  Making  (MCDM)  Methods 

Multi-criteria  decision  making  is  a  well-known  part  of  the  decision  making 
process  and  is  an  important  part  of  the  subject  of  operations  research  when  dealing  with 
more  than  one  decision  criteria  to  make  a  decision.  Conflicts  and  incomparability 
(because  of  not  being  measurable)  of  some  criteria  are  the  problems  that  these  techniques 
are  mostly  facing.  There  are  several  different  methods  under  MCDM’s  umbrella.  Each 
method  has  its  own  characteristics  and,  therefore,  each  method  can  be  the  best  method 
when  the  conditions  suit  its  characteristics.  However,  in  this  study,  as  stated  before,  the 
aim  is  not  to  provide  guidance  for  picking  up  the  best  technique  for  that  special  occasion 
(condition)  or  a  list  of  best  occasions  for  each  technique.  Likewise,  most  of  the  literature 
studies  reviewed  by  the  authors  simply  provide  reasons  why  they  think  that  the  technique 
they  choose  fits  the  best  for  them. 

a.  Weighted  Sum  Method  (WSM)  and  Weighted  Product  Method 
(WPM) 

The  WSM  is  one  of  the  most  popular  and  commonly  used  approaches, 
especially  for  single  dimensional  problems.  When  there  are  M  alternatives  and  N  criteria, 
in  this  method,  the  best  alternative  can  be  found  by  the  following  expression;59 

N 

7=1 

where  A*WSm  is  the  weighted  sum  of  the  best  alternative,  ay  is  the  actual  value  of  ith 
alternative  for  the  jh  criterion,  and  Wj  is  the  weight  of  relative  importance  of  the  jth 
criterion.  The  final  value  of  each  alternative  is  equal  to  the  sum  of  values  for  each 

59  S.  D.  Pohekar  and  M.  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making  to 
Sustainable  Energy  Planning-A  Review,”  Renewable  and  Sustainable  Energy  Reviews  8  (2004):  368, 
http://web.nuu.edu.tw/~ctchen/download/96  l/m/96-05,pdf  (accessed  December  25,  2007). 
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criterion.  This  simple  method  can  have  problems  when  it  is  used  for  multi-dimensional 
problems  and  there  are  different  units;  it  is  meaningless  to  try  to  sum  these  results  in 
different  units.60 


Since  there  is  more  than  one  criterion  and  different  relative  importance  for 
each  criterion,  weights  are  used  to  capture  the  relative  importance  of  the  different  criteria 
to  the  decision  maker.  This  given  relative  importance  of  the  criteria  provides  the  model 
with  tradeoffs,  and  with  the  help  of  weights,  the  model  allocates  their  contribution  to  the 
overall  score.  In  a  scoring  model,  the  weighted  sum  of  the  scores  represents  the  project's 
overall  value  (score). 

While  using  this  technique,  it  is  necessary  to  first  draw  a  hierarchy  tree  of 
objectives/factors  and  then  start  from  the  bottom  level.  Factors  can  have  some  subfactors 
and  subfactors  can  have  elements.  Therefore,  an  overall  score  will  be  calculated  for  each 
subfactor  group  under  a  factor.  Whenever  an  overall  value  is  gotten  from  a  sublevel,  the 
procedure  is  started  from  the  beginning  for  the  new  level.  After  getting  an  overall  value 
(score)  for  the  alternative,  the  same  procedure  is  implemented  for  the  next  alternative. 
After  getting  an  overall  score  for  each  alternative,  the  best  option  will  be  the  alternative 
with  the  highest  score  for  the  decision  maker’s  preferences. 

The  weighted  sum  method,  which  is  also  called  multi-criteria  scoring  or 
the  congruence  model  in  different  literary  sources,  is  a  basic  and  well-known  technique 
that  is  used  in  different  subject  areas  with  sometimes  different  names.  As  it  is  used  in 
MCDM  (with  three  different  names  -  weighted  sum  method,  multi-criteria  scoring  or 
congruence  model),  it  is  also  commonly  used  in  economics  under  'cost-benefit  analysis 
for  measure  of  effectiveness'  calculations.  This  technique  is  also  used  in  statistics  and 
mathematics  as  the  weighted  average  (A  method  of  computing  a  kind  of  arithmetic  mean 
of  a  set  of  numbers  in  which  some  elements  of  the  set  carry  more  importance  (weight) 
than  others).61 


60  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making.” 

61  Cliff  T.  Ragsdale,  Spreadsheet  Modeling  and  Decision  Analysis:  A  Practical  Introduction  to 
Management  Science  (Mason:  South-Western  Learning,  2004),  806. 
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A  very  similar  method  to  WSM  is  the  weighted  product  method  (WPM). 
The  main  difference  between  these  two  methods  is  that  in  WSM,  one  adds  the  values  for 
each  criterion  to  get  the  final  value,  while  in  WPM,  multiplication  of  these  values  take 
place  instead  of  addition.  In  WPM,  each  alternative  is  compared  with  others  by 
multiplying  the  ratios  created  by  each  criterion.  When  trying  to  compare  the  alternatives 
by  using  WPM,  one  takes  Ak  and  Al  as  two  of  the  alternatives  and  solves  the  following 
expression;62 

N  WJ 

R(AK  1  Al)  =  Yj(aki  '  aL,) 

7=1 

where  N  shows  the  number  of  criteria,  a kj  and  ay  are  the  actual  value  of  alternative  K  and 
L  for  the  fh  criterion,  and  Wj  is  the  weight  of  the  relative  importance  of  jth  criterion.  When 
R(Ak/Al)  is  greater  than  1,  A K  is  more  preferable  than  AL  (in  maximization  problems). 
The  overall  best  alternative  is  the  one  which  is  more  preferable  than  every  other 
alternative.63 


b.  Analytical  Hierarchy  Process  (AHP) 

The  analytical  hierarchy  process  (AHP),  developed  by  Thomas  L.  Saaty  at 
the  Wharton  Business  School,  provides  a  tool  to  decision  makers  to  analyze  decisions 
hierarchically  with  the  overall  goal  of  the  decision  at  the  top  of  the  model,  strategic 
objectives  in  the  higher  levels,  and  sub-objectives  with  evaluation  criteria  at  the  bottom.64 
The  hierarchy  is  fairly  simple  but  the  mathematics  is  a  little  complex.  To  start  the 
process,  a  goal  is  broken  down  into  objectives  and  sub-objectives,  as  shown  in  Figure  2, 
until  each  sub-objective  can  be  defined  using  one  or  more  measurable  criteria. 


62  Ragsdale,  Spreadsheet  Modeling,  369. 

63  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  369. 

64  Michael  C.  Weinlein,  “Funding  for  First  Responders  from  a  Threat  and  Prevention  Approach,” 
(master’s  thesis,  Naval  Postgraduate  School,  2004),  22. 
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Goal 


Objective 


Objective 


Objective 


Sub  Sub 

objective  objective 


Sub  Sub 

objective  objective 


Sub  Sub 

objective  objective 


Figure  2.  AHP’s  Hierarchical  Approach. 


Once  the  hierarchy  is  established,  weights  are  assigned  to  every  branch  in 


the  tree.  To  do  this,  analysis  starts  at  the  bottom  of  the  tree.  Starting  with  each  sub¬ 
objective,  the  measurable  criteria  for  that  sub-objective  are  compared  using  a  pair-wise 
comparison  method  such  that  each  criterion  is  compared  against  every  other  criterion, 
including  itself.  This  creates  a  matrix  of  values  that  indicates  the  relative  importance  of 
every  criterion  versus  every  other  criterion  under  a  sub-objective.  This  matrix 


a2\  0-22  '  ’ '  ^2/1 


A  = 


V 


is  created  by  putting  a  numerical  value  resulting  from  the  result  of  the  pair-wise 
comparison  of  criterion  i  with  criterion  j  into  the  position  ap,  and  its  inverse  into  position 
ay,  to  obtain  all  of  the  pair-wise  comparisons  perfonned  for  a  given  sub-objective.65  The 
values  that  are  used  as  entries  in  this  matrix,  and  their  verbal  descriptions,  are  given  in 


Table  6. 


65  Weinlein,  “Funding  for  First  Responders. 
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Relative  Importance 

Definition 

1 

Equally  important 

3 

Moderately  more  important 

5 

Strongly  more  important 

7 

Very  strongly  more  important 

9 

Extremely  more  important 

2, 4, 6, 8 

Intermediate  judgment  values 

Table  6.  Fundamental  Scale  for  Pair-Wise  Comparisons. 


Once  the  matrix  is  determined,  the  values  within  each  column  of  the 
matrix  are  then  nonnalized  so  the  entries  within  each  column  sum  to  one.  Then,  the 
values  within  each  row  are  averaged  to  achieve  a  normalized  vector  of  weights  for  the 
criteria  (the  values  of  this  vector  sum  to  one).  This  vector  of  weights  can  now  be  used  to 
evaluate  alternatives  by  summarizing  the  criteria  values  of  each  alternative  into  a  single 
number  in  a  similar  way  as  the  weighted  sum  method.  This  process  is  repeated  for  each 
level  of  the  hierarchy  so  the  evaluation  of  each  alternative  can  be  summarized  in  a  single 
number  for  final  comparison.  The  alternative  with  the  highest  final  value  should  be  taken 
as  the  best  alternative. 

AHP  also  includes  the  calculation  of  an  inconsistency  index.  This  allows 
the  decision  maker  to  detennine  if  his  or  her  choices  of  pair-wise  comparison  scores  are 
consistent.  Whenever  these  choices  are  consistent,  it  means  that  the  final  decision  is  made 
well.66  A  numerical  example  of  the  analytical  hierarchy  process  can  be  found  in 
Weinlein,  pp.  22-24. 


c.  Outranking  Methods 

When  there  are  not  many  alternatives  to  be  evaluated  in  a  complex 
problem,  which  has  many  criteria  to  be  considered  and  many  participants  involved, 
outranking  methods  can  be  used  by  decision  makers. 


66  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  369. 
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If  the  following  two  conditions  are  met,  it  is  defined  that  the  action  a k 
outranks  the  action  ai: 

•  “ak  is  at  least  as  good  as  ai  with  respect  to  a  major  subset  of  the 
criteria, 

•  ak  is  not  too  bad  relative  to  ai  with  respect  to  the  remaining 
criteria.”67 

The  outranking  methods  demonstrate  the  dominance  of  one  alternative 
over  another  by  using  ordinal  and  descriptive  information.  An  important  feature  they 
have  is  that  they  provide  limited  preference  ranking,  not  a  full  ranking.  They  do  not 
require  a  certain  utility  function  that  shows  the  exact  preference  structure  of  decision 
makers.  They  just  need  enough  information  to  indicate  that  one  alternative  is  better  than 
the  other  one.68 

There  are  two  phases  in  every  outranking  method: 

•  “The  construction  of  an  outranking  relation, 

•  The  exploitation  of  this  relation  in  order  to  assist  the  decision¬ 
maker.”69 


The  most  common  outranking  methods  are  the  preference  ranking 
organization  method  for  enrichment  evaluation  (PROMETHEE)  and  the  elimination  and 
choice  translating  reality  (ELECTRE)  method.70 

Both  methods  involve  a  set  of  alternatives  and  decision  criteria  that  are 
used  as  basis  for  the  evaluations.  These  criteria  can  be  descriptive  or  ordinal  and  may 
involve  uncertainty.  The  uncertainty  in  the  criteria  is  handled  with  a  threshold  model  that 
involves  indifference  and  preference  thresholds.  An  indifference  threshold  is  the  point 


67  “Multi-criterion  decision-making  using  ELECTRE,”  University  of  Geneva,  http://ecolu- 
info.unige.ch/~haurie/mutateAlutate  final/Lectures/Lect  1  3  2/lect  1  3  2.htm  (accessed  December  28, 
2007). 

68  Annika  Kangas,  Jyrki  Kangas,  and  Jouni  Pykalainen,  “Outranking  Methods  As  Tools  in  Strategic 
Natural  Resources  Planning,”  Silva  Fennica  35,  no.  2  (2001);  216, 

http://www.meda. fi/ silvafennica/ full/sO 5/sf3522 1 5 .pdf  (accessed  December  29,  2007). 

69  J.  P.  Brans  and  Ph.  Vincke,  “A  Preference  Ranking  Organization  Method:  (The  PROMETHEE 
Method  for  Multiple  Criteria  Decision-Making),”  Management  Science  31,  no.  6  (1985);.  648, 
http://links.istor.org/sici7sicE0025- 1909(1 98506)3  l%3A6%3C647%3AAPROM(%3E2.Q.CO%3B2-B 

(accessed  December  29,  2007). 

70  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  365. 

34 


below  which  the  two  alternatives  are  indifferent  to  the  decision  maker  and  the  preference 
threshold  is  the  point  above  which  an  alternative  is  dominant  and  preferable  over  another 
alternative.  Between  these  two  thresholds  there  is  a  zone  in  which  the  decision  maker  can 
not  decide  and  it  is  called  the  zone  of  weak  preference.  However,  in  the  case  where  the 
criteria  are  ordinal  or  descriptive,  this  zone  is  not  taken  into  consideration.71 

(1)  Preference  Ranking  Organization  Method  for  Enrichment 
Evaluation  (PRO  ME  THEE).  This  method  needs  a  preference  function  for  each  criterion 
to  compute  the  degree  of  preference.72  “The  credibility  of  the  outranking  relation  that 
alternative  ak  is  better  than  alternative  aj  is  described  by  the  outranking  degree  and  it  is 
calculated  as; 

nK’a/)  =  ZwtF;K’a/) 

j= 1 

where  Fj(ak,  ai)  is  the  preference  function  and  wj  is  the  relative  importance  of  the  different 
criteria.”73 

The  value  of  preference  functions  is  calculated  using  thresholds  pj 

and  qj  as  74 


Fj(ak,ai)  =  < 


1  ^ ifgj  (ak)~ gj  ( a/ ) -  P 
0) tfg j  (ak )-gj(al)<q 
gjM-gjM 


otherwise 


Pi  1 


71  Kangas,  Kangas,  and  Pykalainen,  “Outranking  Methods,”  217. 

72  Jean-Pierre  Brans  and  Bertrand  Mareschal,  “How  to  Decide  with  PROMETHEE,” 
VisualDecision.com.  2,  http://www.visualdecision.com/Pdf/How%20to%20use%20PRQMETHEE.pdf 
(accessed  December  26,  2007). 

73  Kangas,  Kangas,  and  Pykalainen,  “Outranking  Methods,”  218. 

74  Ibid. 
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The  outranking  degree  that  is  calculated  using  this  value  of 
preference  function  and  relative  importance  is  then  used  to  calculate  the  positive, 
negative,  and  net  preference  flows  as  shown  below:75 

\*k 

\*k 

<P{ak)  =  f{.ak)-<j)-(ak) 

The  positive  flow  shows  how  much  an  alternative  is  more  powerful 
than  the  other  alternatives  while  the  negative  flow  shows  how  much  weaker  it  is.76  An 
alternative  outranks  the  other  one  if  its  net  flow  is  higher  than  that  of  the  other  one.  On 
the  other  hand,  if  their  net  flows  are  equal,  they  are  indifferent  to  the  decision  maker. 
Therefore,  the  alternative  that  has  the  highest  net  flow  is  considered  to  be  the  best 
choice.77 

(2)  The  Elimination  and  Choice  Translating  Reality  (ELECTRE). 
When  decision  criteria  are  either  quantitative  or  qualitative  and  not  even  comparable,  this 
outranking  method  can  be  used  to  order  all  alternatives.78  In  the  ELECTRE  method,  the 
concordance  and  discordance  indices  as  well  as  the  threshold  values  are  used  to  draw 
graphs  for  strong  and  weak  relationships.  After  that,  alternatives  are  ordered  in  a  repeated 
process  with  the  assistance  of  these  graphs.79 

An  index  of  concordance  that  demonstrates  that  an  alternative  is  at 
least  as  good  as  another  one  is  calculated  for  each  pair  of  alternatives.  The  formula  used 
to  compute  the  concordance  index  is  shown  below: 

p 

C{ak,aI)  =  YJwjcj(ak,aI) 
j= i 


75  Kangas,  Kangas,  and  Pykalainen,  “Outranking  Methods.” 

76  Brans  and  Mareschal,  “How  to  Decide  with  PROMETHEE,”  2. 

77  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  371. 

78  University  of  Geneva,  “Multi-criterion  decision-making  using  ELECTRE.” 

79  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  371. 
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where  Cj(ak,ai)  is  the  local  concordance  index  and  Wj  is  the  relative  importance  of  the 
different  criteria.80 

The  value  of  the  local  concordance  index  is  calculated  using 
thresholds  pj  and  q,  as81 


Fj(ak,a,)  =  < 


Ufgj{ak)-g]{al)>p 
(X  tfg ,  {ak  )-  gj  (a,  )  -  q 

Sj  {ak)-gj  (a/)"^ 


Pi 


,  otherwise 


As  a  result,  binary  outranking  relations  between  the  alternatives  are 
provided  with  this  method.82 


d.  The  Technique  for  Order  Preference  by  Similarity  to  Ideal 
Solutions  (TOPSIS) 

TOPSIS  was  developed  as  an  alternative  to  ELECTRE  by  Kwangsun 
Yoon  and  Hwang  Ching-Lai  in  1980.  The  basic  concept  of  this  method  is  that  the  most 
preferred  alternative  should  not  only  have  the  shortest  distance  from  the  positive  ideal 
solution  (the  closest  to  positive  ideal),  but  also  have  the  longest  distance  from  the 
negative  ideal  solution  (the  farthest  to  negative  ideal).  This  technique  has  been  widely 
used  in  various  multi-criteria  decision  making  models  due  to  its  simplicity  and 
comprehensibility  in  concept,  its  computational  efficiency,  and  its  ability  to  measure  the 
relative  perfonnance  of  the  decision  alternatives  in  a  simple  mathematical  form.83 


80  Kangas,  Kangas,  and  Pykalainen,  “Outranking  Methods,”219. 

8 '  Kangas,  Kangas,  and  Pykalainen,  “Outranking  Methods.” 

82  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  372. 

83  Yu-Hern  Chang  and  Chung-Hsing  Yeh,  “Evaluating  Airline  Competitiveness  Using  Multi-Attribute 
Decision  Making,”  Omega  the  International  Journal  of  Management  Science  29  (2001);  409, 
http://www.elsevier.com/locate/dsw  (accessed  January  4,  2008). 
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The  assumption  of  the  method  is  that  each  attribute  has  a  monotonically 
increasing  or  decreasing  utility  which  makes  it  easy  to  locate  the  positive  and  negative 
ideal  solutions.  The  preference  order  of  alternatives  is  found  by  comparing  the  Euclidean 
distances  of  alternatives  to  the  ideal  solutions  (positive  and  negative).  A  decision  matrix 
of  M  alternatives  and  N  criteria  is  formulated  firstly.  Then  the  decision  matrix  is 
normalized  and  the  weighted  decision  matrix  is  constructed.  Next,  the  ideal  and  negative 
ideal  solutions  are  found.  The  decision  maker  seeks  to  have  maximum  values  for  the 
benefit  criteria  and  minimum  values  for  the  cost  criteria  among  the  alternatives.  Then, 
separation  measures  are  calculated,  and  finally  relative  closeness  of  each  alternative  to 
the  ideal  solution  is  determined.  The  best  alternative  (the  alternative  with  the  highest 
rank)  is  the  one  which  has  the  shortest  distance  to  the  ideal  solution  and  the  longest 
distance  to  negative  ideal  solution.84 

e.  Compromise  Programming  (CP) 

Compromise  Programming  (CP)  is  a  multiple  criteria  decision  making 
(MCDM)  approach  introduced  by  Yu  and  Zeleny  in  the  1970s.  The  main  idea  behind  this 
approach  is  to  determine  a  subset  of  efficient  solutions  (compromise  set)  that  are  nearest 
to  an  ideal  and  infeasible  point  (ideal  point)  in  which  all  the  criteria  are  optimized.85  It 
defines  the  best  solution  as  the  one  whose  point  is  the  least  farthest  from  the  ideal  point  in 
the  set  of  efficient  solutions.  Finding  a  solution  that  is  as  close  as  possible  to  ideal  is  the 
aim  of  the  approach.  The  distance  measure  used  in  CP  is  the  family  of  Lp-metrics 
formulated  below:86 

^(a)  =  Z<|//-/(a)|/K-wy| 

7=1 

where  Lp  (a)  is  the  Lp  metric  of  alternative  a,/ (a)  is  the  value  of  alternative  a  for  criterion 
j,  Mj  and  are  the  maximum  (ideal)  value  and  the  minimum  (anti  ideal)  value  of 

84  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  372. 

85  Francisco  J.  Andre  and  Carlos  Romero,  “On  the  Equivalence  between  Compromise  Programming 
and  the  Use  of  Composite  Compromise  Metrics,”  (working  paper,  series:  WP  ECON  06.33,  2006),  2 
http://ideas.repec.Org/s/pab/wpaper.html  (accessed  December  25,  2007). 

86  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  372. 
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criterion  j  in  set  A ,  j)  *  is  the  ideal  value  of  criterion  j,  Wj  is  the  weight  of  the  criterion  j, 
and  p  is  the  parameter  reflecting  the  attitude  of  the  decision  maker  with  respect  to 
compensation  between  deviations.  For  p  =  1,  all  deviations  from  jj*  are  taken  into 
account  in  direct  proportion  with  their  assigned  weights.87 

f.  Multi-Attribute  Utility  Theory  (MAUT) 

Multi-attribute  utility  theory  is  another  popular  method  for  decision¬ 
making  that  is  used  to  detennine  the  best  alternative  among  many  and  takes  into 
consideration  the  decision  maker’s  preferences  in  the  form  of  the  utility  function  which  is 
defined  over  a  set  of  attributes.  In  this  method,  all  single  attributes  of  an  alternative  (a 
product  or  a  service)  are  evaluated  to  reach  a  final  decision.88 

It  involves  the  following  set  of  methods  to  make  the  best  decision: 

•  “Define  the  alternatives  and  relevant  attributes. 

•  Evaluate  each  alternative  on  each  attribute.  Remove  dominated 
alternatives. 

•  Assign  relative  weights  to  the  attributes. 

•  Combine  the  attribute  weights  and  evaluations  to  yield  an  overall 
evaluation  of  each  alternative. 

•  Perform  sensitivity  analysis  and  make  a  decision.”89 

The  final  value  of  an  alternative  is  calculated  by  using  the  following 

formula:90 

n 

vM=Ew/v/M 

i= 1 


87  Pohekar  and  Ramachandran,  “Application  of  Multi-Criteria  Decision  Making,”  372. 

88  Bobbi  Blaser  Johnson,  Eli  Gratz,  Kathy  Longenecker  Rust,  and  Roger  Smith,  “Multiattribute  Utility 
Theory,”  (technical  report.  University  of  Wisconsin), 

http://www.r2d2. uwm.edu/atoms/archive/technicalreports/fieldscans/tr-mau.html#maut  at  outcomes 

(accessed  December  11,  2007). 

89  “Multi-Attribute  Utility  Theory,”  University  of  Illinois, 
http://pennmush.tinvmush.org/~alansz/courses/mdm-block/maut.html  (accessed  December  11,  2007). 

90  Ralph  Schafer,  “Rules  for  Using  Multi-Attribute  Utility  Theory  for  Estimating  a  User’s  Interests,” 
Distributed  System  Institute,  3,  http://www.kbs.uni- 

hannover.de/~henze/ABIS  Workshop2001/final/Schaefer  fmal.pdf  (accessed  December  16,  2007). 
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where  vfx)  is  the  value  (utility)  function  of  an  alternative  against  the  i'h  value  dimension 
and  Wi  is  the  weight  assigned  to  that  value  dimension.  The  sum  of  the  all  weights 
assigned  to  each  value  dimension  equals  to  one  and  vfx)  is  calculated  by  using  relevant 
attributes  of  the  value  dimensions  as91 

Vi{X)=TjWaiVai{lia)) 

aeAj 

Here,  A,  is  the  set  of  all  attributes  relevant  for  di,  vai  (1(a))  is  the  evaluation 
of  the  actual  level  1(a)  of  attribute  a  on  di,  and  wai  is  the  weight  that  detennines  the 
impact  of  the  evaluation  of  attribute  a  on  value  dimension  di.  The  variable  wai  is  also 
called  the  relative  importance  of  attribute  a  for  di.  For  all  di  (i=l,...,n)  holds  I  1  * 

3.  An  Alternative  Approach  to  Multi-Criteria  Decision  Making 
Techniques:  Decision  Support  Systems  (DSS)  and  Expert  Systems 

In  some  cases,  decision  makers  do  not  have  the  adequate  background  knowledge 
to  be  able  to  understand  and  use  the  techniques  which  are  briefly  summarized  above. 
Instead,  these  decision  makers  can  use  some  software  tools,  which  are  built  to  use  the 
techniques  (one  or  some  of  them)  with  a  user  friendly  interface,  to  get  the  support  to 
make  a  decision.  These  support  tools  are  decision  support  systems  and  expert  systems. 

“Decision  Support  Systems  (DSS)  are  interactive  information  systems  that  rely  on 
an  integrated  set  of  user-friendly  decision  support  tools  to  produce  and  present 
information  to  support  management  in  the  decision  making  process.”92 

Most  of  the  time,  managers  rely  on  their  personal  experience  and  the  infonnation 
that  is  available  to  help  them  in  making  decisions,  but  sometimes  when  encountering  a 
real  complex  decision,  their  human  abilities  are  not  capable  of  evaluating  for  the  best 93 


91  Schafer,  “Rules  for  Using  Multi -Attribute  Utility  Theory,”  3. 

92  Larry  Long  and  Nancy  Long,  Computers  Information  Technology  in  Perspective  (New  Jersey: 
Pearson  Education  Inc.  Press,  2005),  375. 

93  Ibid. 
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When  it  is  a  complex  decision,  DSS  helps  decision  makers  to  choose  between 
alternatives  and  can  even  rank  alternatives  based  on  the  user’s  preferences.  Also,  DSS 
can  help  managers  to  close  the  gap  between  the  necessary  information  and  available 
information.94 

Decision  support  systems  generally  consist  of  a  database,  a  knowledge  base,  a 
model  base,  and  a  user  interface.  The  database  provides  the  relevant  historical  data  for  the 
system  to  use  in  the  model  base.  The  knowledge  base  keeps  the  rules  that  are  constraining 
the  solution  alternatives.  The  model  base  includes  statistical,  forecasting,  simulation,  and 
other  math  tools.  The  model  base  also  provides  users  with  appropriate  tools  without 
developing  a  model  from  the  beginning.  The  user  interface  allows  users  to  control  the 
DSS  easily  and  help  them  to  decide  which  data  and  techniques  to  use  in  the  analysis.95 

The  most  important  characteristics  of  DSS  are  helping  decision  makers,  especially 
at  tactical  and  strategic  levels,  in  addressing  semi-structured  and  unstructured  problems, 
being  interactive  and  user  friendly,  and  being  adaptable  to  meet  the  requirements  of  any 
decision.96 

Data  management,  modeling,  statistical  analysis,  planning,  inquiry,  representing 
information,  and  consolidation  of  similar  information  are  the  most  important  and 
common  capabilities  of  DSS.97 

Another  type  of  information  systems  is  Expert  Systems  (ES).  “An  Expert  System 
is  an  interactive  system  that  responds  to  questions,  asks  for  clarification,  makes 
recommendations,  and  generally  helps  the  users  in  decision  making  process.”98  Expert 
systems  are  the  simulation  of  the  human  thinking  process,  and  it  is  like  working  with  a 


94  Long  and  Long,  Computers  Information,  375. 

95  Spainhower,  “An  Exploratory  Study,”  42. 

96  Long  and  Long,  Computers  Information,  375. 

97  Ibid.,  376. 

98  Ibid.,  379. 


41 


human  expert  to  solve  a  problem.  But,  expert  systems  are  better  because  they  do  not  miss 
any  important  consideration  or  alternative.  Expert  systems  are  using  if-then  step-by-step 
rules  to  solve  the  problems." 

C.  CONCLUSION 

Multi-criteria  decision  analysis  (MCDA)  offers  numerous  methodologies  and 
schools  of  thought  for  multi-criteria  modeling  and  decision  support,  as  the  authors 
attempted  to  discuss  the  most  popular  ones  in  this  chapter.  The  approaches  vary  from 
very  simple  ones  to  very  complex  optimization  models. 

The  main  aim  of  this  study  is  to  show  the  subjectivity  involved  in  offeror 
(proposal)  evaluation  in  the  source  selection  part  of  government  acquisitions.  Therefore, 
the  authors  preferred  to  choose  a  technique  that  is  simple  to  understand  by  anyone  who 
does  not  have  a  multi-criteria  decision  making  (MCDM)  background.  WSM  is  one  of  the 
very  basic  models/techniques  used  in  situations  where  more  than  one  criterion  is  involved 
and  the  relative  importance  of  each  criterion  is  different  from  each  other.  The  model’s 
biggest  strength  is  its  simplicity  in  application. 

In  the  last  thirty  years,  very  different  methods  were  developed  in  the  area  of 
MCDM.  Most  of  the  improvements  are  related  with  the  setting  of  priorities  (relative 
importance).  Whenever  the  number  of  criteria  increases  and  subcriteria  exists,  providing 
consistency  of  the  relative  importance  becomes  harder  and  harder.  As  stated  before,  this 
technique  is  one  of  the  very  basic  methods;  it  is  not  providing  a  decision  maker  with  a 
tool  to  determine  the  weights  consistently.  The  decision  maker  has  the  responsibility  to 
provide  the  consistency  which  is  especially  needed  while  assigning  weights  that  is 
supposed  to  reflect  their  relative  importance. 


"  Long  and  Long,  Computers  Information. 
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In  this  chapter,  an  overview  on  the  available  multi-criteria  decision  support  tools 
and  techniques  was  given.  Among  all  these  techniques,  in  this  study  one  of  the  basic 
methods  will  be  used  to  help  the  authors  show  the  aim  of  their  study.  The  next  chapter 
will  discuss  the  Air  Force’s  KC-X  program  in  which  the  authors  will  use  a  simplified 
version  of  it  in  their  study  as  a  sample  case. 
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IV.  USAF’S  KC-X  NEXT  GENERATION  TANKER  PLANES 

PROGRAM 


A.  INTRODUCTION 

In  this  part  of  the  study,  the  authors  will  give  some  background  information  about 
the  Air  Force’s  KC-X  program.  The  authors  will  use  this  program  in  their  model  to  show 
the  subjectivity  involved  in  the  source  selection  proposal  evaluation  part  of  Air  Force’s 
acquisitions.  First,  some  introductory  information  about  the  program  will  be  given,  and 
then,  the  requirements  will  be  explained  in  moderate  detail.  Next,  background  on  how  the 
competition  began  between  two  offerors  will  be  discussed,  and  lastly,  some  specific 
information  (to  roughly  compare  the  two  alternatives)  about  each  proposed  aircraft  will 
be  presented. 

The  aim  of  the  authors’  project  is  to  show  the  effects  of  the  subjectivity  involved 
in  government  procurement  processes.  Subjectivity  involvement  in  the  evaluation  of  the 
proposals  in  the  source  selection  part  of  an  acquisition  is  inevitable.  However, 
subjectivity  contrasts  with  fairness  and  best  value.  The  authors  try  to  show  the  effects  of 
subjectivity  involved  in  the  process  of  selection  by  using  a  simplified  and  current  real-life 
program. 

The  authors  chose  the  USAF’s  KC-X  program  to  study  based  upon  their  research. 
There  are  some  specific  reasons  for  the  authors  to  choose  that  program.  First  of  all,  they 
wanted  to  show  the  effects  of  subjectivity,  so  they  decided  to  choose  an  Air  Force 
procurement  program  in  which  color  ratings  were  used.  This  is  because  in  the  authors’ 
perspective  the  Air  Force’s  color  rating  is  one  of  the  most  subjective  techniques  used  in 
the  government  source  selection. 

The  authors  decided  to  study  a  program  that  was  popular  and  current,  as  it  was 

thought  that  they  could  have  lots  of  different  comments  from  a  multitude  of  viewpoints. 

With  a  current  program,  they  hoped  to  have  more  insight  information,  which  was 

publicly  available,  about  the  program.  Otherwise,  most  of  the  infonnation  about  old 

programs  is  not  accessible  online  because  of  confidentiality  issues.  Also,  it  was  very  clear 
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that  they  would  not  be  able  to  get  any  official  infonnation  about  an  ongoing  program  but 
they  thought  that  they  could  get  enough  data  to  do  this  study  from  the  publicly  available 
part  of  the  data  about  the  program.  After  all  these  considerations,  they  came  up  with  the 
idea  of  doing  their  study  on  USAF’s  KC-X  program. 

After  doing  some  research  about  the  program,  they  realized  that  the  two 
alternatives  offered  very  different  specifications  and  capabilities  for  the  same  program. 
This  showed  the  authors  the  difference  of  the  two  offeror’s  point  of  view.  The  authors 
thought  that  this  program  might  be  a  good  sample  for  their  study  because  of  the  potential 
effects  of  subjectivity. 

Another  factor  that  made  the  authors’  choice  even  more  interesting  for  the  study 
is  the  high  potential  of  the  program  to  get  protested.  Experts  of  the  defense  industry  were 
expecting  a  protest  regardless  of  the  winner.  Since  the  proposed  KC-X  alternatives  have 
very  different  strengths  in  terms  of  capabilities,  results  would  not  easily  satisfy  the 
offeror  who  lost  the  program  award. 

The  acquisition  of  weapon  systems  has  always  been  a  crucial  part  of  government 
acquisitions.  They  are  crucial  not  only  because  of  the  dollar  amount  that  goes  along  with 
these  acquisitions,  but  also  because  the  final  product  will  be  an  important  determinant 
factor  in  national  security.  Also,  these  programs  take  too  much  time  and  effort. 
Consequently,  these  projects  may  keep  the  U.S.  as  a  super  power  and  pioneer  new 
technologies  when  they  have  successfully  ended,  but  may  keep  the  U.S.  behind  when 
they  are  not  successful.  There  are  other  important  factors  that  have  impact  on  these 
processes  like  not  having  enough  independent  offerors  and  competition.  Therefore,  these 
projects  are  necessary  but  risky. 

The  Department  of  Defense  (DOD)  and  the  Air  Force  have  become  increasingly 
concerned  about  their  aging  aerial  refueling  aircraft  because  of  the  heavy  pace  of 
operations  in  support  of  the  war  on  terrorism.  The  aerial  refueling  fleet  was  last  reviewed 
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by  the  Government  Accountability  Office  (GAO)  in  1996  and  it  was  found  that  KC-135 
aircraft  were  aging  and  becoming  increasingly  costly  to  maintain  and  operate.100 

Currently,  one  of  the  biggest  weapon  systems  projects  is  the  U.S.  Air  Force’s  KC- 
X  program.  In  January  2007,  the  solicitation  for  the  KC-X  project  was  advertised  on  the 
Federal  Business  Opportunities  (FBO)  website.  The  program  has  been  planned  for  the 
production  of  175  airplanes  and  4  test  platforms  (i.e.,  a  total  of  179  airplanes).  The  cost 
objective  for  the  first  phase  was  approximately  $40  billion.  When  the  program  was 
announced,  the  USAF  declared  their  thoughts  about  renewing  the  Air  Force’s  tanker  fleet 
(America’s  current  aerial  tanker  fleet  is  approximately  40-50  year  old)  project  as  their 
number  one  priority.101 

The  releases  stated  that  “the  Air  Force  also  intends  to  take  full  advantage  of  the 
other  capabilities  inherent  in  the  platform,  and  make  it  an  integral  part  of  the  Defense 
Transportation  System.”  Contending  aircrafts  were  offering  substantial  improvements 
over  the  KC-135’s  extra  capacity  for  cargo  or  people,  in  addition  to  their  tanker  roles.102 

Lt.  Gen.  Donald  Hoffman,  the  Assistant  Secretary  of  the  Air  Force  for 
Acquisition,  announced  the  posting  of  the  KC-X  Aerial  Refueling  Aircraft  RFP  to  the 
FBO  website  on  January  30,  2007,  signaling  the  official  launch  of  the  Air  Force’s  number 
one  priority  acquisition  program.  The  release  noted  that  “The  KC-X  program  is  the  first 
of  three  acquisition  programs  the  Air  Force  will  need  to  replace  the  entire  fleet  of  aging 
KC-135  Stratotankers,  which  have  been  in  service  for  more  than  50  years”.  The  RFP 
stipulated  nine  primary  key  performance  parameters:103 

1 .  Air  refueling  capability 

2.  Fuel  offload  and  range  at  least  as  great  as  the  KC-135 

100  U.S.  General  Accounting  Office  (GAO),  MILITARY  AIRCRAFT:  DOD  Needs  to  Determine  Its 
Aerial  Refueling  Aircraft  Requirements,  GAO-04-349  Report  June  2004, 
http://www.gao.gov/new.items/d04349.pdf  (accessed  March  1 1,  2008). 

101  August  Cole,  “Airbus  to  Raise  Ante  in  Bid  for  Military-Tanker  Deal”,  Wall  Street  Journal,  January 
14,  2008,  6,  http://ebird.afis.mil/ebfiles/e20080114572969.html.  (accessed  January  21,  2008). 

102  “jhe  USAF’s  KC-X  Aerial  Tanker  RFP,”  http://www.defenseindustrvdailv.com/the-usafs-kcx- 
aerial-tanker-rfp-03 009/  (accessed  February  1,  2008). 

103  “The  USAF’s  KC-X  Aerial  Tanker  RFP.” 
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3.  Compliant  communication,  navigation,  surveillance/air  traffic 
management  (CNS/ATM)  equipment 

4.  Airlift  capability 

5.  Ability  to  take  on  fuel  while  airborne 

6.  Sufficient  force  protection  measures 

7.  Ability  to  network  into  the  information  available  in  the  battlespace 

8.  Survivability  measures  (defensive  systems,  Electro-Magnetic  Pulse  (EMP) 
hardening,  chemical/biological  protection,  etc,) 

9.  Provisioning  for  a  multi-point  refueling  system  to  support  Navy  and  Allied 
aircraft 

The  USAF  said  that  final  RFP  defined  an  integrated,  capability-based,  best-value 
approach,  and  included  specific  factors  for  assessing  the  capability  contribution  of  each 
offeror,  along  with  cost  and  assessments  of  past  performance  and  proposal  risk. 

Also,  it  was  announced  by  the  Air  Force  that  the  department  had  gone  through  a 
careful  review  process  for  KC-X  and  had  validated  that  the  RFP  had  accurately  reflected 
the  requirements  as  laid  out  by  the  warfighter. 

The  Boeing  707-based  KC-135  fleet  ranges  in  age  from  40  to  50  years  old,  raising 
the  risk  that  fatigue  or  aging-related  problems  could  ground  them  at  some  unanticipated 
time.  Since  aerial  transport,  fighter  strike  missions,  bomber  missions,  and  combat  air 
patrols  all  depend  on  aerial  refueling  to  some  degree,  a  grounding  of  the  KC-135  fleet 
could  be  catastrophic  for  America’s  military  posture.104 

The  primary  mission  of  the  KC-X  will  be  to  provide  aerial  refueling  to  the  United 
States  military  and  coalition  aircrafts  in  the  war  on  terrorism  and  other  missions. 
However,  the  Air  Force  also  intends  to  take  full  advantage  of  the  other  capabilities 
inherent  in  the  platform  and  make  it  an  integral  part  of  the  Defense  Transportation 
System.  The  RFP  identified  nine  primary  key  performance  parameters:  addressing  air 
refueling  capabilities,  including  fuel,  receiving  fuel  by  aerial  refueling  from  other 


104  “USAF  KC-X:  Will  There  Be  a  Competition?”  http://www.defenseindustrvdailv.com/usaf-kcx- 
will-there-be-a-competition-02994/  (accessed  February  1,  2008). 
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platforms,  and  range  at  least  equal  to  those  of  the  existing  KC-135.  A  new  capability 
addressing  joint  operations  will  be  the  provisioning  for  a  multi-point  refueling  system,  to 
support  Navy  and  allied  aircraft.105 

B.  USAF’S  REQUIREMENTS  FOR  KC-X 

According  to  the  Headquarters  Air  Mobility  Command  White  Paper  on  KC-X, 
Joint  Doctrine  is  the  root  of  the  requirement  for  a  flexible  Aerial  Refueling  Aircraft  that 
can  operate  throughout  a  battlespace  to  deliver  fuel  and/or  cargo  and/or  passengers.  The 
Nation’s  new  KC-X  aircraft,  which  is  required  to  be  equipped  with  appropriate  floors  for 
carrying  passengers  and  cargo,  reasonably-sized  doors  to  accommodate  standard-sized 
pallets,  and  modest  defensive  systems  which  allow  the  aircraft  access  to  an  area  of 
operations,  will  help  US  Combatant  Commanders  to  success  in  their  mission.106 

Also,  this  document  states  that,  US  Joint  Publications  recognized  the  need  for  a 
refueling  tanker  to  be  able  to  manage  different  types  of  missions  at  the  same  time  and 
have  transferred  these  valuable  lessons  into  current  doctrine.  For  example,  JP  3-17,  one 
of  the  joint  doctrines  related  with  air  mobility,  states  that  all  USAF  tanker  aircrafts  are 
capable  of  performing  an  airlift  role  and  are  used  to  augment  core  airlift  assets.  Dual  role 
concept  demands  an  air  refueling  aircraft  that  can  transport  a  combination  of  passengers 
and  cargo  while  performing  air  refueling.  The  same  concept  states  that  in  some 
circumstances,  it  may  be  more  efficient  employing  an  air  refueling  aircraft  strictly  in  an 

airlift  role.  Air  refueling  units’  deployment  because  of  their  organic  capacity  may  be 
more  effective  to  transport  unit  personnel  and  support  equipment  or  passengers  and  cargo 
from  other  units.107 


105“US  Air  Force's  KC-X  Aerial  Tanker  Replacement  Program.”  http ://www. defense- 
update.  com/newscast/0 107/news/300 107  kcx.htm  (accessed  February  1,  2008). 

106  Fleadquarters  Air  Mobility  Command  White  Paper,  “KC-X:  The  Next  Mobility  Platform  the  Need 
for  a  Flexible  Tanker.”  http://www-tc.pbs.org/newshour/bb/military/tanker  kc-x.pdf  (accessed  March  11, 
2008). 

107  Fleadquarters  Air  Mobility  Command  White  Paper,  “KC-X:  The  Next  Mobility  Platform  the  Need 
for  a  Flexible  Tanker.” 
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The  USAF  announced  its  final  solicitation  (RFP)  on  January  30,  2007,  and  in  its 
RFP,  the  USAF  explained  the  requirements  for  the  KC-X.  Mission  capability,  proposal 
risk,  past  perfonnance,  cost/price,  and  the  integrated  fleet  aerial  refueling  assessment 
(IFARA)  will  be  the  live  main  factors  for  the  USAF’s  best  value  product  evaluation  for 
this  program.  The  USAF  explained  this  in  their  RFP  numbered  FA8625-07-R-6470  and 
dated  January  30,  2007,  as  stated  below, 

The  Government  will  select  the  best  overall  offer,  based  upon  an  integrated 
assessment  of  Mission  Capability,  Proposal  Risk,  Past  Performance, 
Cost/Price  and  the  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA). 
Contract(s)  may  be  awarded  to  the  offeror  who  is  deemed  responsible  in 
accordance  with  the  FAR,  as  supplemented,  whose  proposal  conforms  to  the 
solicitation's  requirements  (to  include  all  stated  terms,  conditions, 
representations,  certifications,  and  all  other  information  required  by  Section  L 
of  the  solicitation)  and  is  judged,  based  on  the  evaluation  factors  and 
subfactors,  to  represent  the  best  value  to  the  Government.  The  Government 
seeks  to  award  to  the  offeror  who  gives  Air  Force  the  greatest  confidence  that 
it  will  best  meet,  or  exceed,  the  requirements.  This  may  result  in  an  award  to 
a  higher  rated,  higher  priced  offeror,  where  the  decision  is  consistent  with  the 
evaluation  factors  and  the  Source  Selection  Authority  (SSA)  reasonably 
determines  that  the  technical  superiority  and/or  overall  business  approach 
and/or  superior  past  performance,  and/or  the  IFARA  of  the  higher  priced 
offeror  outweighs  the  cost  difference.  The  SSA  will  base  the  source  selection 
decision  on  an  integrated  assessment  of  proposals  against  all  source  selection 
criteria  in  the  solicitation  (described  below).  While  the  Government  source 
selection  evaluation  team  and  the  SSA  will  strive  for  maximum  objectivity, 
the  source  selection  process,  by  its  nature,  is  subjective  and,  therefore, 
professional  judgment  is  implicit  throughout  the  entire  process.  108 

The  USAF  will  make  the  award  to  the  best  valued  option.  The  evaluation  will  be 
made  as  explained  in  the  RFP  Section  M.  The  main  factors  are  given  in  the  order  of 
importance  (mission  capability,  proposal  risk,  past  performance,  cost/price  and  IFARA) 
and  for  the  USAF,  the  mission  capability,  proposal  risk,  and  past  performance  evaluation 
factors  are  of  equal  importance  and  the  cost/price  and  IFARA  evaluation  factors  are  also 
of  equal  importance.  The  first  three  evaluation  factors  are  more  important  than  the  other 
two  factors.  The  USAF  explained  this  in  their  RFP  as  stated  below, 


108  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  1. 
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Award  will  be  made  to  the  offeror  submitting  the  most  advantageous 
proposal  to  the  Government  based  upon  an  integrated  assessment  of  the 
evaluation  factors  and  subfactors  described  below.  The  Mission 
Capability,  Proposal  Risk,  and  Past  Performance  evaluation  factors  are  of 
equal  importance  and  individually  more  important  than  either  Cost/Price 
or  IFARA  evaluation  factors  individually.  The  IFARA  is  equal  in 
importance  to  Cost/Price.  Within  the  Mission  Capability  factor,  the  five 
(5)  subfactors  are  listed  in  descending  order  of  relative  importance  from  1 
to  5.  In  accordance  with  FAR  15.304(e),  the  Mission  Capability,  Proposal 
Risk,  Past  Performance,  and  IFARA  evaluation  factors,  when  combined, 
are  significantly  more  important  than  Cost/Price;  however,  Cost/Price  will 
contribute  substantially  to  the  selection  decision.109 

Each  of  the  five  factors  will  be  discussed  separately  to  give  more  specific 
information  about  each  of  them  and  how  they  should  be  evaluated. 

1.  Factor  1:  Mission  Capability 

The  offeror's  capability  to  satisfy  the  government's  requirements  will  be  assessed 
by  the  mission  capability  evaluation.  This  factor  has  5  subfactors  to  evaluate.  All  mission 
capability  subfactors  will  each  receive  one  of  the  color  ratings  described  in  Air  Force 
Federal  Acquisition  Regulation  Supplement  (AFFARS)  in  Mandatory  Procedures  part 
MP5315.3,  paragraph  5.5.1  (blue-exceptional,  green-acceptable,  yellow-marginal,  red- 
unacceptable).  “Assessment  will  focus  on  the  strengths  and  deficiencies  of  the  offeror's 
proposal.  The  color  rating  represents  how  well  the  offeror's  proposal  meets  the  Mission 
Capability  subfactor  requirements.”  Mission  Capability  subfactor  5  will  only  receive  1  of 

the  following  3  ratings:  acceptable  (green),  marginal  (yellow),  or  unacceptable  (red)). 
There  will  not  be  any  overall  color  rating,  which  is  rolled  up  from  subfactor  ratings,  for 
the  mission  capability  factor.110 

a.  Subfactor  1:  Key  System  Requirements 

The  first  subfactor  under  mission  capability  is  key  system  requirements. 
The  offeror’s  understanding  of  the  requirements  (as  defined  in  the  system  requirements 


109  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  2. 

110  Ibid.,  3. 
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document  (SRD))  and  how  its  ability  to  meet  those  requirements  has  substantiated  will  be 
evaluated  under  this  subfactor.  Only  the  logistics  requirements  as  addressed  in  the 
Product  Support  section  of  subfactor  3  will  not  be  evaluated  herein  subfactor  1 . 1 1 1 

The  government’s  evaluation  of  the  offeror's  approach  to  meet  SRD 
requirements,  according  to  the  USAF’s  solicitation  (RFP)  document,  numbered  FA8625- 
07-R-6470,  explained  below. 

Aerial  refueling  will  include  tanker  aerial  refueling,  receiver  aerial 
refueling,  fuel  offload  versus  radius  range,  drogue  refueling  systems  (including 
simultaneous  multipoint  refueling),  the  operationally  effective  size  of  the  boom  envelope, 
the  aerial  refueling  operator  station  and  aircraft  fuel  efficiency.  Airlift  will  include  airlift 
efficiency,  cargo,  passengers,  aero-medical  evacuation,  ground  turn  time,  and  cargo  bay 
re-configuration.  Operational  Utility  will  include  aircraft  maneuverability,  worldwide 
airspace  operations,  communication/information  systems  (including  Net-Ready 
capability),  treaty  compliance  support,  formation  flight,  intercontinental  range,  7,000  foot 
runway  operations,  bare  base  airfield  operations,  and  growth  provisions  for  upgrades. 
Survivability  will  include  situational  awareness,  defensive  systems  against  threats, 
chemical/biological  capability,  EMP  protection,  fuel  tank  fire/explosion  protection,  and 
night  vision  capability.  Other  system  requirements  will  include  all  other  SRD 
requirements  that  are  not  above  (in  Aerial  Refueling,  Airlift,  Operational  Utility  and 
Survivability)  or  in  Subfactor  3. 112 

b.  Subfactor  2:  System  Integration  and  Software 

The  second  subfactor  under  mission  capability  is  system  integration  and 
software.  This  subfactor  will  evaluate  the  offeror’s  ability  to  implement  disciplined  and 
institutionalized  systems  engineering  approach  and  the  offeror's  capability  to  manage  and 
integrate  the  software  elements  to  satisfy  performance  capability  requirements  in  the  KC- 
X’s  SRD.  The  USAF  explained  this  in  their  RFP  as  stated  below.113 

111  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  3. 

112  Ibid.,  6. 

113  Ibid. 
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The  Government  will  evaluate  the  proposal  to  determine  the  offeror's 
ability  to  implement  a  disciplined  and  institutionalized  systems 
engineering  approach  necessary  to  successfully  design,  develop,  integrate, 
validate  and  verify  requirements,  manufacture,  and  sustain  the  KC-X 
system  as  defined  by  the  perfonnance  capability  requirements  set  forth  in 
the  KC-X  SRD.  The  software  development  capability  (SDC)  will  be 
evaluated  to  detennine  the  offeror's  capability  to  manage  and  integrate  the 
software  elements  required  to  satisfy  the  performance  requirements.114 


c.  Subfactor  3:  Product  Support 

Another  subfactor  under  mission  capability  is  product  support.  The 
offeror's  proposed  product  support  approach  will  be  evaluated  for  an  efficient,  effective, 
and  comprehensive  support  program  for  the  service  life  of  the  KC-X  fleet.  The  offeror's 
approach  to  achieve  reliability,  availability,  maintainability,  and  supportability  with  an 
optimal  logistics  footprint  will  be  evaluated  by  the  government  under  this  subfactor. 
Operational  availability  (Ao),  reliability  and  maintainability  (R&M),  and  mission 
capability  (MC)  rate  will  be  the  detennination  points  of  the  government’s  evaluation.115 

d.  Subfactor  4:  Program  Management 

The  program  management  approach  that  the  offeror  will  use  for  the  KC-X 
program  is  also  another  important  part  of  the  mission  capability  factor.  It  is  the  fourth 

subfactor  of  mission  capability.  The  government  will  evaluate  the  offeror’s  proposal  to 
determine  if  the  offeror  is  using  a  realistic  and  reasonable  approach  to  effectively  and 
efficiently  implement  and  manage  the  KC-X  program.116 

The  USAF  explained  how  the  realism  and  reasonableness  determinations 
will  be  done  in  their  RFP  as  stated  below. 

Realism  will  be  assessed  to  ensure  the  offeror's  proposal  reflects  a  clear 
understanding  of  program  requirements,  correlates  to  other  program 
documentation,  and  is  consistent  with  the  approach  described  in  the 


114  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  5. 

115  Ibid.,  6. 

116  Ibid.,  7. 
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technical  volumes.  Reasonableness  will  be  assessed  to  ensure  the 
proposed  logic  and  methodology  reflected  in  program  documentation  is 
acceptable  and  reflects  an  understanding  of  commonly  accepted  program 
management  concepts  and  practices  (emphasis  added).117 

e.  Subfactor  5:  Technology  Maturity  and  Demonstration 

The  last  of  the  subfactors  of  mission  capability  is  technology  maturity  and 
demonstration,  and  this  subfactor  will  evaluate  the  maturity  of  the  critical  technology 
elements  (CTE)  that  are  included  in  the  offeror's  proposed  KC-X  aircraft.  These  critical 
technology  elements  are  described  in  the  DoD  Technology  Readiness  Assessment  (TRA) 
Deskbook  that  can  be  accessed  from  the  DoD’s  website.  A  CTE  is  an  element  that  is  new 
or  novel  or  is  being  used  in  a  new  or  novel  way.  It  is  critical  if  it  is  necessary  to  achieve 
the  successful  development  of  a  system,  its  acquisition,  or  its  operational  utility.  For 
example,  a  few  of  these  can  be  mechanical  components,  processors,  servers,  electronics, 
and  software  (algorithm)  interfaces.  But,  with  the  guidance  provided  by  the  DoD 
Technology  Readiness  Assessment  (TRA)  Deskbook,  these  elements  are  assessed  for 
each  program  specifically.118 

2.  Factor  2:  Proposal  Risk 

A  proposal  risk  factor  evaluation  will  be  done  at  the  mission  capability  subfactor 
level  for  only  subfactors  1-4.  The  weaknesses  associated  with  an  offeror's  proposed 
approach  will  be  focused  on  while  doing  the  proposal  risk  evaluation.  This  evaluation 
will  try  to  determine  the  risks  involved  in  program  management  objectives  like  cost, 
scheduling,  and  perfonnance.  An  assessment  of  the  potential  for  disruption  of  schedule, 
increased  cost,  degradation  of  perfonnance,  and  the  need  for  increased  government 
oversight,  as  well  as  the  likelihood  of  unsuccessful  contract  perfonnance,  will  be  the  key 
points  of  this  evaluation.119 


117  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  7. 
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The  first  four  mission  capability  subfactors  will  receive  one  of  the  proposal  risk 
ratings  (high,  moderate,  low)  described  in  AFFARS  MP5315.3,  paragraph  5.5.2.  The 
risks  related  with  the  weaknesses  or  significant  weaknesses  of  the  offeror's  proposed 
approach  to  each  of  the  mission  capability  subfactors  will  be  the  focus  of  the  evaluation. 
If  any  weakness  is  identified,  then  the  evaluation  will  also  focus  on  the  offeror's  proposed 
improvements  and  whether  that  approach  is  manageable  or  not. 

3.  Factor  3:  Past  Performance 

The  degree  of  confidence  the  government  has  in  an  offeror's  ability  to  provide  the 
product  or  service  that  meets  warfighters'  needs,  including  cost  and  schedule,  is 
detennined  by  the  past  performance  evaluation  factor  and  based  on  records  of 
demonstrated  performance.120 

The  Perfonnance  Confidence  Assessment  Group  (PCAG)  is  the  government 
evaluation  team  that  will  conduct  an  in-depth  review  and  evaluation  of  all  obtained 
performance  data.  The  PCAG  will  work  on  both  past  and  present  perfonnance  data 
identified  by  offerors  in  their  proposals  and  additional  past  and  present  perfonnance  data, 
if  available  from  other  sources.  If  necessary,  the  PCAG  will  also  confirm  the  past  and 
present  performance  data  identified  by  offerors  in  their  proposals.  Additionally,  the 
offeror's  perfonnance  in  managing  and  mitigating  program  risk  will  be  assessed.  After  an 
evaluation  of  the  offeror's  recent  past  performance  relevant  to  only  mission  capability 
subfactors  1-4  and  cost/price,  the  government  will  assess  an  overall  performance 
confidence.  The  overall  past  performance  evaluation  results  will  be  the  government's 
confidence  in  the  offeror's  ability  to  fulfill  the  solicitation  requirements  while  meeting 
schedule,  budget,  and  performance  quality  constraints.121 

To  also  understand  the  government’s  evaluation  in  a  better  way,  some  notions  like 
performance  confidence  assessment,  relevance,  and  offerors  without  a  record  of  relevant 


120  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  9. 
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past  performance  and  performance  problems  should  be  understood  correctly.  The  AF 
explained  these  in  their  RFP  as  stated  below, 

Performance  Confidence  Assessment  -  under  the  past  perfonnance 
factor,  the  perfonnance  confidence  assessment  represents  the  evaluation  of 
an  offeror’s  present  and  past  work  record  in  order  to  assess  the 
government's  confidence  in  the  offeror's  probability  of  successfully 
performing  as  proposed  (emphasis  added).122 

The  perfonnance  confidence  assessment  will  be  assessed  at  the  overall 
factor  level  after  evaluating  aspects  of  the  offeror's  recent  past 
performance,  focusing  on  performance  that  is  relevant  to  the  mission 
capability  subfactors  1-4  only  and  cost  or  price....  Each  offeror  will 
receive  one  of  the  confidence  ratings  (high  confidence,  significant 
confidence,  satisfactory  confidence,  unknown  confidence,  little 
confidence,  or  no  confidence)  as  prescribed  in  AFFARS  MP  5315.305 
(August  10,  2005). 123 

Relevance  -  the  past  performance  evaluation  is  accomplished  by 
reviewing  aspects  of  an  offeror's  and  major/critical  subcontractor's 
relevant  present  and  recent  past  performances  (emphasis  added).124 

Offerors  without  a  record  of  relevant  past  perfonnance  or  for  whom 
information  on  past  performance  is  not  available  will  not  be  evaluated 
favorably  or  unfavorably  based  on  past  perfonnance  and,  as  a  result,  will 
receive  an  "unknown  confidence"  rating  for  the  past  performance  factor 
(emphasis  added).125 

Where  relevant  perfonnance  records  indicate  performance  problems,  the 
government  will  consider  the  number  and  severity  of  the  problems  as  well 
as  the  appropriateness  and  effectiveness  of  any  conective  actions  taken 
(not  just  planned  or  promised).  The  government  may  review  more  recent 
contracts  or  performance  evaluations  to  ensure  corrective  actions  have 
been  implemented  and  evaluate  their  effectiveness  (emphasis  added).126 


122  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  10. 
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4. 


Factor  4:  Cost/Price 


The  cost/price  evaluation  will  be  done  by  using  criteria  to  evaluate  the  current 
program  costs,  then  the  most  probable  life  cycle  cost  (MPLCC)  evaluation  will  be  done, 
and  finally  proposals  will  be  analyzed  to  detennine  whether  or  not  they  are  unbalanced 
with  respect  to  prices,  quantity  matrix  factors,  or  separately  priced  line  items. 

Criteria,  MPLCC  and  unbalanced  pricing  are  the  important  points  of  the  cost/price 
evaluation  and  the  USAF  explained  these  in  their  RFP  as  stated  below. 

•  Criteria  -  the  cost  panel  will  evaluate  the  offeror's  cost  proposal  against 
realism,  reasonableness,  and  cost/price  risk  rating.127 

•  Most  Probable  Life  Cycle  Cost  (MPLCC)  is  an  independent  government 
cost  estimate,  which  is  adjusted  for  technical,  cost,  and  schedule  risks,  to 
include  all  contract,  budgetary,  and  other  government  costs.  Costs  those 
are  associated  with  all  phases  of  the  entire  weapon  system  life  cycle  will 
be  included.  A  work  breakdown  structured  approach  will  be  used  to  come 
up  with  an  independent  government  cost  estimate.128 

•  Unbalanced  Pricing  -  proposals  will  be  analyzed  to  detennine  whether  or 
not  they  are  unbalanced  with  respect  to  prices,  quantity  matrix  factors,  or 
separately  priced  line  items.  Unbalanced  pricing  exists  whenever  the  total 
solicitation  effort  price  is  acceptable  but,  the  price  of  one  or  more  contract 
line  items  or  a  factor  in  the  quantity  matrices  is  significantly  overstated  or 
understated  (can  be  demonstrated  by  the  application  of  price  analysis 
techniques  in  FAR  15.404-1  (b)). 129 

5.  Factor  5:  Integrated  Fleet  Aerial  Refueling  Assessment 

The  integrated  fleet  aerial  refueling  assessment  (IFARA)  factor  evaluation  will  be 
done  by  using  a  scenario-based  simulation  program.  An  offeror’s  KC-X  fleet  data  from 
its  proposal  will  be  evaluated  on  a  specific  scenario  by  this  simulation  program  to 
determine  the  number  of  KC-X  to  meet  peak  demand  for  mission  accomplishment.  The 
same  scenario  will  evaluate  the  number  of  KC-135R  needed  to  accomplish  the  mission 
instead  of  KC-X.  For  comparison,  the  number  of  KC-135R  needed  for  the  mission  will  be 


127USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  12. 
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divided  by  the  number  of  KC-X  needed.  This  ratio  will  be  called  the  fleet  effectiveness 
value  and  will  show  the  effectiveness  of  KC-X  instead  of  KC-135R.  A  fleet  effectiveness 
value  of  1.0  will  be  assessed  as  equal  in  effectiveness  to  the  KC-135R,  so  a  fleet 
effectiveness  value  greater  than  1.0  will  be  assessed  as  more  effective  than  the  KC-135R 
and  more  advantageous  to  the  government.  The  USAF  explained  the  IFARA  factor  in 
their  RFP  as  stated  below, 

The  Government  will  use  modeling  and  simulation  to  provide  an 
integrated  assessment  of  the  utility  and  flexibility  for  a  fleet  of  the 
offeror's  proposed  KC-X  by  evaluating  the  number  of  aircraft  required  to 
fulfill  the  peak  demand  of  the  aerial  refueling  elements  evaluated  in  the 
2005  Mobility  Capability  Study  (MCS).  In  the  context  of  this  evaluation 
scenario,  the  Government  will  determine  the  proposed  KC-X's  fleet 
effectiveness  in  relation  to  a  KC-135R  fleet.130 

The  Government  will  conduct  the  analysis  using  offeror-provided  data  in 
the  evaluation  scenario  by  primarily  using  the  Combined  Mating  and 
Ranging  Planning  System  (CMARPS)  modeling  and  simulation  tool.  The 
results  of  the  CMARPS  evaluation  will  provide  the  Government  with  the 
quantity  (based  on  the  offeror-proposed  KC-X  aircraft)  required  to  meet 
the  mission  requirements  of  the  evaluation  scenario.  The  same  scenario 
will  be  run  on  CMARPS  using  a  KC-135R  fleet  to  provide  a  baseline 
quantity  for  comparison.  The  required  number  of  KC-135R  aircraft 
generated  by  the  model  will  be  divided  by  the  number  of  proposed  KC-X 
aircraft  required  to  meet  the  same  scenario.  This  ratio  is  the  "fleet 
effectiveness  value"  for  the  proposed  KC-X  aircraft.131 

The  Government  will  report  the  "fleet  effectiveness  value"  as  detennined 
by  the  evaluation  as  a  standalone  "value"  to  the  SSA,  along  with  any 
major  insights  and  observations  gleaned  from  the  evaluation.  This  value 
will  be  detennined  by  Government  analysis  taking  into  account  the 
offeror's  input  data  and  considering  any  analysis  performed  by  the  offeror 
of  the  same  evaluation  scenario.132 


130  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  14. 
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c. 


BOEING  AND  NORTHROP  GRUMMAN-EADS  TEAM  COMPETED  FOR 
KC-X  PROGRAM 


Shortly  after  the  AF  released  its  RFP,  Boeing  and  Northrop  Grumman-European 
Aeronautic  Defense  and  Space  Company  (EADS)  announced  that  they  would  bid  on  the 
program,  and  the  two  teams  began  competing  for  this  contract  award.  On  one  side  was 
the  Boeing  team  with  KC-767  Advanced  (767-200  derivative)  and  on  the  other  was  the 
Northrop  Grumman-EADS’s  team  with  KC-30B  (Airbus  A330-200/200F  derivative). 
Approximately  a  year  after  solicitation,  final  proposals  were  submitted  on  January  3, 
2008. 133 

Both  teams  waited  hopefully  for  the  results  which  were  publicly  announced  on 
February  29,  2008.  The  Northrop  Grumman-EADS  (Airbus)  team  was  announced  as  the 
winner  of  this  program  and  it  was  a  surprise  for  lots  of  people  who  were  expecting 
Boeing  to  win  (Airbus  is  an  aircraft  manufacturing  subsidiary  of  EADS). 

The  announcement,  available  on  DOD’s  website  about  the  USAF’s  aerial  tanker 
program  award,  stated  the  contract  award  as  follows; 

Northrop  Grumman  Corp.,  of  Los  Angeles,  Calif.,  is  being  awarded  a  cost 
plus  incentive/award  fee,  fixed  price  incentive,  firm  fixed  price  contract 
for  the  newly-named  KC-45.  This  contract  is  awarded  after  full  and  open 
bidding,  and  provides  for  the  system  design  and  development  of  four  test 
aircraft  for  $1.5B.  This  contract  also  includes  five  production  options 
targeted  for  64  aircraft  at  $10.6B.At  this  time  no  funds  have  been 
obligated.  Contracting  activity  is  the  Aeronautical  Systems  Center, 
Wright-Patterson  Air  Force  Base,  Ohio  (contract  number  FA8625-08-C- 
645 1).134 

On  March  7,  2008,  Air  Force  and  Boeing  officials  came  together  for  a  formal 
debriefing.  After  a  thorough  analysis  of  data  presented  at  this  debriefing  on  the  decision, 
Boeing  concluded  that  the  Air  Force’s  efforts  to  run  a  fair,  open,  and  transparent 
competition  failed  due  to  irregularities  that  placed  Boeing  at  a  competitive  disadvantage 


133  “The  USAF’s  KC-X  Aerial  Tanker  RFP.” 

' 3 ^  U.S.  Department  of  Defense,  “Contract  Announcement  webpage,” 
http://www.defenselink.mil/contracts/contract.aspx7contracticE3719  (accessed  March  1,  2008). 
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throughout  this  competition.  Boeing  claimed  that  because  of  these  irregularities  they  were 
penalized  for  offering  a  commercial-derivative  airplane  with  lower  costs  and  risks  and 
greater  protection  for  troops.135 

On  March  11,  2008,  Boeing  filed  a  formal  protest  to  the  Government 
Accountability  Office  (GAO)  claiming  that  there  were  irregularities  with  the  process  of 
the  competition  and  the  evaluation  of  the  competitors’  bids  and  cited  these  irregularities, 
which  are  not  fully  available  to  the  public,  in  their  formal  protest.  The  protest  was  not  a 
surprise  for  defense  industry  experts  because  of  the  huge  difference  between  the  offered 
products  of  the  two  competing  offerors  for  the  same  requirements.  Thus,  experts  were 
expecting  a  protest  from  the  losing  party  (either  Boeing  or  NG-Airbus).  In  the  protest, 
Boeing’s  main  point  was  the  size  of  NG-Airbus’s  KC-X  since  the  AF  formerly 
announced  this  program  as  a  replacement  program  for  the  KC-135  and  also  announced 
that  there  would  be  another  program  to  replace  KC-lOs  (KC-10  is  an  aircraft  that  is 
almost  two  times  larger  than  the  KC-135).  The  NG-Airbus  team  proposed  an  aircraft  that 
was  even  bigger  than  the  KC-10  for  the  KC-135  replacement  program,  so  Boeing  has 
asked  the  agency  to  review  the  USAF’s  decision  to  award  a  contract  to  the  Northrop 
Grumman  and  EADS  team  to  replace  aerial  refueling  tankers.136 

Boeing’s  Vice  President  and  Tanker  Program  Manager,  Mark  McGraw,  said  that 
their  analysis  of  the  data  presented  by  the  Air  Force  showed  that  the  competition  was 
seriously  flawed  and  resulted  in  the  selection  of  the  wrong  airplane  for  the  warfighters. 
He  also  talked  about  their  concerns  about  the  USAF’s  evaluation  and  expressed  that  they 
were  exercising  their  protesting  process.  The  GAO  will  review  the  decision  to  ensure  that 
the  process  was  fair  and  the  results  were  the  best  choice  for  the  U.S.  warfighters  and 
taxpayers.137 


135  Boeing,  “Boeing  Protests  U.S.  Air  Force  Tanker  Contract  Award,” 
http://www.boeing.com/news/releases/20Q8/ql/08031  lb  nr.html  (accessed  March  12,  2008). 

136  Boeing,  “Boeing  Protests  U.S.  Air  Force  Tanker  Contract  Award.” 

137  Ibid. 
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D. 


INFORMATION  ABOUT  KC-X  ALTERNATIVES 


The  aim  of  this  project  is  to  show  the  effects  of  subjectivity  involved  in  the  source 
selection  phase  of  acquisition  because  of  the  techniques  used  in  proposal  evaluations.  The 
authors  try  to  show  the  effects  of  subjectivity  by  using  the  KC-X  program,  but  they  do 
not  try  to  make  the  proposal  evaluation  for  the  KC-X  program.  Even  if  they  wanted  to  do 
the  evaluation  for  proposals,  they  would  need  lots  of  experts  and  much  time  like  the  Air 
Force  did.  Since  the  authors  want  to  work  on  realistic  data,  they  use  the  data  about  the 
program  that  was  publicly  available.  The  authors  expect  to  have  more  realistic  results  this 
way.  All  the  information  the  authors  used  were  from  offerors’  websites  for  this  specific 
program  and  the  product  cards  of  the  proposed  aircrafts.  Also,  they  have  another  source 
of  infonnation,  which  is  the  publicly  available  part  of  Boeing’s  protest  file.  The  authors 
work  on  the  KC-X  program  with  this  publicly  available  infonnation. 

Since  the  entire  program’s  goal  is  to  replace  the  aging  KC-135s  with  their  new 
alternative,  all  mission  capability  requirements  include  the  KC-135  as  the  base  point. 
Therefore,  the  authors  provide  some  very  basic  capabilities  of  the  KC-135  in  this  study, 
which  will  be  very  helpful  for  one  to  understand  the  USAF’s  comparisons  as  well  as  the 
capabilities  of  the  new  alternatives  in  a  better  way. 

While  the  KC-135s  can  carry  up  to  6  standard  463F  cargo  pallets,  53  people,  or 
about  18  medical  litters,  according  to  the  KC-30  team’s  (Northrop  Grumman-EADS) 
official  brochure,  their  A330-200/200F  derivative  can  carry  up  to  32  standard  463F  cargo 
pallets,  up  to  226  passengers,  or  120  medical  litters,  or  some  combination  of  the  above,  in 
addition  to  its  full  fuel  load.  Team  Boeing’s  KC-767  Advanced  can  carry  up  to  19 
standard  463L  cargo  pallets,  up  to  190  passengers,  or  54  medical  litters,  or  some 
combination  of  the  above,  in  addition  to  its  full  fuel  load. 

The  exact  amount  of  fuel  that  the  US  KC-767  Advanced  can  carry  is  not  public. 
The  base  figure  of  around  200,000  pounds  is  similar  to  the  existing  KC-135,  but  the 
additional  body  tank  option  is  a  wild  card.  The  KC-767’s  advanced  refueling  boom  has  a 
900  gallon/minute  capacity  and  been  tested  successfully  in  live  air-to-air  refuelings. 
While  the  KC-30  can  carry  over  250,000  pounds  of  capacity,  it  also  carries  more  fuel 
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than  the  161 ,  which  is  a  particular  advantage  in  the  Pacific  sector  with  its  wide  over¬ 
water  expanses.  On  the  other  hand,  its  advanced  1,200  gallon/minute  ARBS  refueling 
boom  has  yet  to  actually  transfer  fuel  to  another  aircraft  in  the  air.  Figure  3  illustrates  the 
size  comparison  of  the  KC-30,  KC-767AT  and  KC-135. 


Figure  3.  Size  Comparison  of  the  KC-30,  KC-767AT  and  KC-135  (Source:  Boeing’s 

KC-767’s  product  card). 

The  KC-30  is  approximately  two  times  larger  than  the  KC-135  and  also 
approximately  one  and  a  half  times  larger  than  the  767,  which  requires  a  slightly  longer 
runway  and  costs  more  to  operate  on  a  per-plane  basis.  Team  KC-30  stresses  costs  and 
efficiency  on  a  per-mission  basis,  and  if  larger  tanker  aircraft  with  more  fuel  and  cargo 
space  mean  fewer  sorties  required,  the  figures  may  look  more  equal  once  the  mission  the 
tankers  are  supporting  is  complete.138 

As  their  relative  capacities  demonstrate,  the  KC-767  is  a  smaller  aircraft  than  the 
KC-30.  One  positive  consequence  is  that  it  can  take  off  from  slightly  shorter  runways. 
The  USAF  requires  the  ability  to  take  off  from  an  8,000  foot  runway,  but  would  prefer 
7,000  feet  as  this  makes  more  runways  available.  Boeing  also  claims  the  KC-767  as  being 

138  “The  USAF’s  KC-X  Aerial  Tanker  RFP.” 
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22  to  24  percent  cheaper  to  operate  and  maintain  on  a  per-plane  basis  than  the  KC-30, 
and  its  base  aircraft  is  cheaper  to  buy  on  the  civilian  market.139 

Figure  4  below  is  the  evaluation  summary  table  taken  from  Boeing’s  protest  file 
(the  publicly  available  part  of  the  protest  file).  These  are  the  evaluation  summary  results 
that  Boeing  claims  that  the  AF  came  up  with  at  the  end  of  its  evaluations  on  proposals.140 


Evaluation  Summary  of  Factors 


Factors 

Mission  CapabHitvfProtKtsal  Risk 

Key  System  Requirements 
System  Integration  &  Software 
Product  Support 
Program  Management 
Technology  Maturity  &  Demonstration 

Performance  Confidence 

Cogj/Prjce 
Reasonableness 
Realistic 
Balanced  Offer 
MPLCC  (TY$) 

Cost  RISK  Rating  (SOD  -  P&D) 

IFARA 


Figure  4.  Evaluation  Summary  Table  taken  from  Boeing’s  Protest  File. 


This  figure  was  removed  from  Boeing’s  publicly  available  protest  file  after  the 
USAF  and  NG  separately  made  objections  against  it  because  it  included  proprietary 
information. 

However,  the  authors  will  use  these  evaluation  results  from  the  evaluation 
summary  table  presented  above.  In  this  study,  the  aim  is  to  show  the  subjectivity  involved 
in  source  selection,  especially  in  the  color  rating  method;  therefore,  the  evaluation  result 
in  this  study  will  not  be  an  alternative  evaluation  for  the  KC-X  program. 


139  “The  USAF’s  KC-X  Aerial  Tanker  RFP.” 

140  Boeing,  Boeing’s  Protest  File  (publicly  available  part), 
http://www.boeing.com/ids/globaltanker/pdf7executive  summary.pdf  (accessed  March  18,  2008). 
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V.  MODEL  CONSTRUCTION 


A.  INTRODUCTION 

In  this  study,  two  models  that  have  been  built  via  the  use  of  Microsoft  Excel  are 
presented  in  two  versions  to  show  different  characteristics  of  two  types  of  methods  that 
can  be  used  in  the  source  selection  phase  in  government  acquisitions.  These  methods  are 
the  color  rating  method,  which  is  the  current  method  used  by  the  Air  Force,  and  the 
quantitative  weighted  sum  method.  In  the  study,  the  authors  have  added  a  numerical 
structure  to  the  color  rating  method  to  be  able  to  perform  their  analysis.  Since  it  is  not 
possible  to  compare  numerical  and  adjective  information  and  make  an  analysis,  the 
authors  have  converted  the  color/adjectival  ratings  into  numerical  points  using  a  way  that 
will  be  thoroughly  explained  in  this  chapter. 

The  models  have  been  established  using  some  data  from  the  Air  Force’s  current 
KC-X  Tanker  Replacement  Program  and  proposals  of  its  two  offerors,  Boeing  and 
Northrop/Airbus . 

In  version  one  of  the  authors’  models,  they  used  exact  color  and  adjectival  ratings 
given  in  the  KC-X  solicitation  and  determined  the  ranges  that  are  used  to  assign 
numerical  points  to  factors  and  subfactors  centered  on  a  basic  logic.  On  the  other  hand,  in 
the  second  version,  the  authors  eliminated  some  color  and  adjectival  ratings  and  changed 
the  structures  of  the  ranges  to  be  able  to  respond  to  some  complicated  needs.  The  details 
about  both  versions  will  be  provided  in  the  authors’  model  building  explanations. 

B.  MODEL  BUILDING  OF  VERSION  ONE 

In  model  building  for  version  one,  the  authors  will  use  a  model  for  color  rating 
evaluations  and  will  use  another  model  for  weighted  sum  method  evaluations.  The  idea 
behind  these  models  is  to  show  the  difference  between  two  techniques  for  the  same 
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purpose  and  also  to  show  how  the  result  will  be  affected.  The  steps  that  compose  the 
version  one  models  with  detailed  explanations  will  start  with  the  color  rating  model  and 
after  finishing  it  completely,  the  weighted  sum  method  model  will  be  explained. 

1.  The  Model  Using  the  Color  Rating  Method 

The  model  building  for  the  color  rating  method  will  consist  of  the  following  eight 

steps: 

1.  Determining  factors  and  sub  factors 

2.  Determining  relative  importance  of  factors  and  subfactors 

3.  Evaluating  the  proposed  systems  and  giving  ratings  to  factors  and 
subfactors 

4.  Assigning  weights  to  factors  and  subfactors  based  on  their  relative 
importance 

5.  Determining  the  ranges  of  ratings  and  the  ranges’  midpoints 

6.  Assigning  the  related  values  for  those  ratings 

7.  Calculating  weighted  ratings  and  final  ratings 

8.  Comparing  both  offeror's  final  ratings  and  deciding  on  the  winner 

a.  Determine  Factors  and  Subfactors 

Factors  and  subfactors  are  the  discriminators  against  which  each  offeror’s 
proposal  is  evaluated.  Factors  without  any  subfactors  as  well  as  factors  with  several 
subfactors  are  possible  in  big  govermnent  acquisitions.  In  the  KC-X  program,  the  factors 
and  sub  factors  were  presented  in  the  solicitation  as  follows: 

•  Factor  1 :  Mission  Capability 

•  Subfactor  1 :  Key  System  Requirements  (KSR) 

•  Subfactor  2:  System  Integration  and  Software  (SIS) 

•  Subfactor  3:  Product  Support  (PS) 

•  Subfactor  4:  Program  Management  (PM) 

•  Subfactor  5:  Technology  Maturity  and  Demonstration 
(TMD) 

•  Factor  2:  Proposal  Risk 

•  Factor  3:  Past  Perfonnance 
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•  Factor  4:  Cost/Price 

•  Factor  5:  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA) 

In  the  proposal  risk  assessment,  the  offerors  are  evaluated  at  the  mission 
capability  subfactor  level  using  the  first  four  subfactors,  which  are  key  system 
requirements,  system  integration  and  software,  product  support,  and  program 
management.  Therefore,  these  four  factors  are  considered  as  subfactors  in  the  proposal 
risk  assessment  in  the  authors’  models. 

Furthermore,  a  cost/price  risk  evaluation  as  well  as  life  cycle  cost  is 
included  in  the  cost/price  factor  assessment.  Hence,  there  are  two  subfactors,  life  cycle 
cost  and  cost  risk,  under  this  factor. 

In  the  integrated  fleet  aerial  refueling  assessment,  the  “fleet  effectiveness 
value”,  which  is  the  standalone  value  that  will  be  reported  to  the  Source  Selection 
Authority  (SSA)  according  to  the  solicitation,  will  be  the  only  consideration  point  on  this 
factor. 

After  these  explanations  the  final  form  for  the  factors  and  subfactors  will 

be  as  follows: 

•  Factor  1:  Mission  Capability 

•  Subfactor  1 :  Key  System  Requirements  (KSR) 

•  Subfactor  2:  System  Integration  and  Software  (SIS) 

•  Subfactor  3:  Product  Support  (PS) 

•  Subfactor  4:  Program  Management  (PM) 

•  Subfactor  5:  Technology  Maturity  and  Demonstration 
(TMD) 

•  Factor  2:  Proposal  Risk 

•  Subfactor  1 :  Key  System  Requirements  (KSR) 

•  Subfactor  2:  System  Integration  and  Software  (SIS) 

•  Subfactor  3:  Product  Support  (PS) 

•  Subfactor  4:  Program  Management  (PM) 

•  Factor  3:  Past  Perfonnance 
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•  Factor  4:  Cost/Price 

•  Subfactor  1 :  Life  Cycle  Cost 

•  Subfactor  2:  Cost  Risk 

•  Factor  5:  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA) 

b.  Determine  Relative  Importance  of  Factors  and  Subfactors 

In  the  KC-X  solicitation,  the  relative  importance  of  factors  and  subfactors 
is  given  using  narrative  explanations  and  terms  like  “significant”,  “more  important,” 
“equal,”  or  “less  important”.  As  an  important  point,  in  the  color  rating  method,  a 
mathematical  differential  between  factors  and  subfactors  using  terms  like  “twice  as 
important  as”  is  not  assigned.141 

The  relative  importance  of  factors  and  subfactors  is  provided  in  the  KC-X 
solicitation  as  follows: 

The  Mission  Capability,  Proposal  Risk,  and  Past  Perfonnance  evaluation 
factors  are  of  equal  importance  and  individually  more  important  than 
either  Cost/Price  or  IFARA  evaluation  factors  individually.  The  IFARA  is 
equal  in  importance  to  Cost/Price.  Within  the  Mission  Capability  factor, 
the  five  (5)  subfactors  are  listed  in  descending  order  of  relative  importance 
from  1  to  5.  In  accordance  with  FAR  15.304(e),  the  Mission  Capability, 
Proposal  Risk,  Past  Performance,  and  IFARA  evaluation  factors,  when 
combined,  are  significantly  more  important  than  Cost/Price;  however, 
Cost/Price  will  contribute  substantially  to  the  selection  decision.142 

As  stated  above,  mission  capability,  proposal  risk,  and  past  performance, 
and  on  the  other  side  cost/price  and  IFARA,  are  equally  important,  whereas  the  first  three 
factors  are  individually  more  significant  than  either  cost/price  or  IFARA.  Also,  the  order 
of  relative  importance  for  the  mission  capability  subfactors  is  as  follows: 

1 .  Key  System  Requirements  (KSR) 

2.  System  Integration  and  Software  (SIS) 

3.  Product  Support  (PS) 


141  Slate,  “Best  Value  Source  Selection,”  52. 

142  USAF’s  FA8625-07-R-6470  numbered  Solicitation  (RFP)  Document,  Section  M,  2. 
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4.  Program  Management  (PM) 

5.  Technology  Maturity  and  Demonstration  (TMD) 

On  the  other  hand,  under  the  cost/price  factor,  it  has  been  assumed  that  life 
cycle  cost  is  more  important  than  the  cost  risk  assessment  when  its  real  importance  as 
stated  in  the  solicitation  is  considered. 

c.  Evaluate  the  Proposed  Systems  and  Assign  Ratings  to  Factors 
and  Subfactors 

In  this  phase  of  the  authors’  model  building,  they  have  mainly  used  the 
data  they  gathered  from  different  sources,  i.e.,  bidders’  web  sites.  In  this  way,  the 
authors’  goal  is  to  maintain  realism  throughout  their  models.  Color  and  adjectival  ratings 
are  assigned  to  factors  and  subfactors  for  each  offeror  based  on  the  merits  of  their 
proposals. 

At  this  point,  it  should  be  stated  that  the  Source  Selection  Authority  (SSA) 
is  supposed  to  make  its  decision  based  on  these  color  and  adjectival  ratings  after  this  step. 
However,  because  it  is  so  complicated  to  make  a  decision  based  on  these  ratings  and 
consistency  is  so  important  while  evaluating  both  offerors,  the  SSA  should  use  a  type  of 
numerical  method.  Based  on  this  logic,  the  authors  have  built  the  rest  of  their  color  rating 
model  using  some  numerical  ways. 

d.  Assign  Weights  to  Factors  and  Subfactors  Based  on  Their 
Relative  Importance 

In  the  authors’  models,  they  have  arbitrarily  assigned  some  specific 
weights  to  factors  and  subfactors  to  be  able  to  reflect  their  real  importance  levels  as  given 
in  the  RFP.  Without  finding  an  appropriate  way  to  reflect  these  importance  levels,  it  is 
impossible  to  be  consistent  while  evaluating  both  offerors,  and  assigning  numerical 
weights  is  the  most  easy  and  non-complex  way  to  accomplish  this  goal.  As  an  example, 
because  mission  capability  is  more  important  than  cost/price  according  to  the  solicitation, 
mission  capability  should  get  a  higher  weight  than  cost/price. 
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On  the  other  hand,  at  each  level  of  factors  and  subfactors,  the  sum  of 
weights  should  be  equal  to  100  to  maintain  consistency.  Figure  5  depicts  the  weights  the 
authors  assigned. 
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Figure  5.  The  Weights  of  the  Factors  and  the  Subfactors. 


e.  Determine  the  Ranges  of  Ratings  and  the  Ranges '  Midpoints 

In  order  to  combine  weights  and  ratings  at  the  last  phase  of  the  evaluation 
and  reach  an  overall  rating,  each  color  rating  and  adjectival  rating  should  get  a  numerical 
value.  Therefore,  in  the  authors’  model  building,  they  have  initially  established  some 
numerical  ranges  that  represent  each  color  and  adjectival  rating  and  are  set  between 
numbers  0  and  100. 

It  was  stated  before  that  any  factor  or  subfactor  except  life  cycle  cost  and 
IFARA,  which  get  numerical  values,  can  get  any  color  or  adjectival  rating  based  on  the 
value  an  offeror  proposes.  In  order  to  show  which  points  between  0  and  100  represent 
which  color  or  adjectival  rating,  a  number  of  ranges  should  be  determined  based  on  the 
number  of  color  or  adjectival  ratings  possible  for  a  factor  or  a  subfactor. 
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In  this  part  of  the  authors’  study,  the  ranges  for  each  color  or  adjectival 
rating  are  set  as  equal.  For  example,  when  there  are  four  color  ratings,  each  color  rating 
will  have  a  range  of  25  points  on  a  100-point  scale.  Choosing  unequal  ranges  is 
investigated  in  an  upcoming  section. 

Additionally,  the  authors  have  to  assign  only  one  numerical  rating  to  each 
color  or  adjectival  rating  and  that  numerical  point  should  be  a  number  among  the 
numbers  in  the  range  that  rating  is  associated  with.  In  this  study,  the  authors  have 
assigned  the  midpoints  of  the  ranges  to  the  color  and  adjectival  ratings  because  a 
midpoint  is  the  most  appropriate  value  that  can  represent  the  color  or  adjectival  rating  in 
a  range. 

The  following  diagrams  in  Figure  6  show  the  ranges  and  the  midpoints  for 

the  sub  factors. 
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Figure  6.  The  Ranges  and  the  Midpoints  of  the  Factors  and  the  Subfactors  for  Version 

One. 
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As  a  last  point,  it  has  been  assumed  that  the  cost  range  is  between  105  and 
110  billion  dollars  because  the  estimated  life  cycle  cost  is  about  108  billion  dollars. 
Additionally,  the  IFARA  range  is  between  0.75  and  2.00  because  the  real  IFARA  values 
of  the  offerors  are  1.79  and  1.90  and  the  authors  wanted  to  take  into  account  the  values 
below  1  in  this  version. 

f  Assign  the  Related  Values  to  the  Ratings 

The  midpoints  determined  before  for  each  numerical  range  were  assigned 
to  the  color  and  adjectival  ratings.  In  this  way,  a  combination  of  these  points  and 
numerical  weights  will  be  possible  to  reach  an  overall  rating  for  an  offeror. 

g.  Calculate  Weighted  Ratings  and  Final  Ratings 

In  this  phase,  weighted  ratings  are  calculated  by  the  multiplication  of 
weights  and  assigned  midpoints.  For  past  performance  and  IFARA,  the  factors  that  do  not 
have  any  subfactors,  only  one  calculation  is  enough  to  reach  the  overall  factor  rating. 
However,  for  other  factors  that  have  subfactors,  one  more  calculation  is  needed  to  reach 
the  overall  factor  rating.  After  having  all  factor  ratings,  an  overall  offeror  rating  is 
obtained  by  using  the  same  calculation. 

The  basic  calculation  used  in  this  step  is  shown  below; 

Factor  or  Subfactor  Rating  =  (Factor  or  Subfactor  Weight)  *  (Assigned  Midpoint) 

Throughout  the  authors’  study,  they  have  taken  advantage  of  the  software 
program  Microsoft  Excel  that  can  make  these  kinds  of  calculations  fast  and  let  users 
easily  change  inputs  to  see  instantaneous  changes  on  outputs.  The  authors  are  going  to 
explain  how  they  have  used  Excel  during  this  phase  of  their  study  on  two  examples. 

Before  any  action,  the  appearance  of  the  spreadsheet  for  the  mission 
capability  subfactors  is  shown  in  Figure  7. 
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Figure  7.  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Mission 
Capability  Subfactor  before  Any  Action  in  Version  One. 


For  the  mission  capability  subfactor  key  system  requirements,  after 
evaluation  of  the  proposed  system  the  user  enters  an  assessed  color  rating  into  cell  C4.  As 
soon  as  the  user  enters  the  rating  into  cell  C4,  Excel  assigns  the  associated  midpoint  into 
cell  F4  (refer  to  Figure  8). 
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Figure  8.  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Mission 
Capability  Subfactor  after  Entering  the  First  Color  Rating  in  Version  One. 


To  be  able  to  assign  the  true  midpoint  associated  with  any  color  rating,  the 
F4  cell  as  seen  in  Figure  8  has  the  following  fonnula: 

=IF(C4="BLUE",87.5,IF(C4="GREEN",62.5,IF(C4="YELLOW",37.5,IF(C4="RED", 

12.5,0)))) 
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This  formula  tells  Excel  to  assign  87.5  if  the  color  rating  is  blue,  62.5  if 
the  color  rating  is  green,  37.5  if  the  color  rating  is  yellow,  12.5  if  the  color  rating  is  red, 
or  0  for  other  entries.  Since  green  has  been  entered  into  cell  C4,  Excel  assigned  62.5  into 
cell  F4.  Also,  the  multiplication  of  weight  (E4)  and  numerical  rating  (F4)  gives  the 
weighted  rating. 


After  entering  all  assessed  color  ratings  into  cells  C4-C8,  Excel  assigns 
related  midpoints  into  cells  F4-F8,  which  have  the  same  formula  above  (except  F8 
because  Technology  Maturity  and  Demonstration  (TMD)  subfactor  does  not  get  a  blue 
rating  and  has  different  midpoints)  with  true  cell  names,  and  all  weighted  ratings  are 
calculated  via  the  scheme  stated  above.  As  a  final  step,  Excel  instantaneously  adds  all 
weighted  ratings  and  the  overall  numerical  rating  for  the  mission  capability  factor,  which 
is  52.08  in  this  example,  is  reached  (refer  to  Figure  9). 
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Figure  9.  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Mission 
Capability  Subfactor  after  Entering  All  Color  Ratings  in  Version  One. 

As  a  second  example,  the  appearance  of  the  spreadsheet  for  the  cost/price 
factor  before  any  entry  is  given  below  in  Figure  10. 
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Figure  10.  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Cost/Price 

Subfactors  before  Any  Action  in  Version  One. 


For  the  cost/price  sub  factor  cost,  the  user  enters  the  life  cycle  cost  the 
offeror  proposes  into  cell  C4.  As  soon  as  the  user  enters  the  cost  data  into  cell  C4,  Excel 
assigns  the  associated  numerical  rating  into  cell  F4  (refer  to  Figure  11). 

Due  to  the  fact  that  the  life  cycle  cost  range  is  between  105  and  110  billion 
dollars,  F4  cell  has  the  following  formula: 

=IF  (C4==0,  0,  2200-(20*C4)) 


This  formula  tells  Excel  to  assign  0  points  into  cell  F4  whenever  there  is 
no  value  assigned  and  whenever  a  value  is  assigned,  the  formula  will  use  the  equation  to 
calculate  the  rating.  For  instance,  if  the  proposed  cost  is  110  billion  dollars,  Excel  will 
assign  0  points  into  cell  F4  and  100  points  if  the  proposed  cost  is  105  billion  dollars. 
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Figure  1 1 .  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Cost/Price 

Subfactors  after  Entering  the  LCC  in  Version  One. 


For  the  cost/price  subfactor  risk,  after  evaluation  of  the  proposed  system, 
the  user  enters  the  assessed  adjectival  rating  into  cell  C5.  As  soon  as  the  user  enters  the 
rating  into  cell  C5,  Excel  assigns  the  associated  midpoint  into  cell  F5  (refer  to  Figure  12 
below). 
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Figure  12.  The  Appearance  of  the  Color  Rating  Method  Spreadsheet  for  the  Cost/Price 
Sub  factors  after  Entering  All  Ratings  in  Version  One. 

To  be  able  to  assign  the  true  midpoint  associated  with  any  adjectival 
rating,  the  F5  cell  has  the  following  fonnula: 

=IF(C5="HIGH", 16.7, IF(C5="MODERATE", 50, IF(C5="LOW", 83.3,0))) 


This  formula  tells  Excel  to  assign  the  value  of  16.7  if  the  adjectival  rating 
is  high,  50  if  the  rating  is  moderate,  83.3  if  the  rating  is  low,  or  0  for  all  other  entries. 
Since  low  has  been  entered  into  cell  C5,  Excel  has  assigned  83.3  to  cell  F5.  Also,  the 
multiplication  of  weight  (E5)  and  numerical  rating  (F5)  gives  the  weighted  rating.  As  a 
final  step,  Excel  instantaneously  adds  all  weighted  ratings,  and  the  overall  numerical 
rating  for  cost/price  factor,  which  is  69.32  in  this  example,  is  obtained. 

Finally,  after  all  factor  ratings  are  obtained  and  weighted  factor  ratings  are 
computed  using  factor  weights  assigned  in  step  (d),  an  overall  offeror  rating  is  reached  by 
adding  all  weighted  factor  ratings.  The  final  appearance  of  an  example  spreadsheet  in 
which  the  overall  offeror  rating  is  58.2025  is  given  below  in  Figure  13. 
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Figure  13.  Overall  Offeror  Ratings  for  Version  One. 
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h.  Compare  Both  Offeror's  Final  Ratings  and  Decide  the  Winner 


As  the  last  phase,  the  two  offerors  are  compared  using  their  overall 
numerical  ratings  and  the  offeror  that  has  higher  point  wins  the  competition.  Thus,  the 
contract  is  awarded  to  that  offeror. 

An  example  snapshot  of  the  version  one  color  rating  spreadsheet  as  a 
whole  is  presented  in  the  Appendix  part  of  the  study. 

2.  The  Model  Using  the  Quantitative  Weighted  Sum  Method 

This  method  consists  of  the  same  steps  like  the  color  rating  method  except  steps 
(e)  and  (f),  because  numerical  ratings  instead  of  color  or  adjectival  ratings  are  used  in  the 
quantitative  weighted  sum  method  and  thus  there  is  no  need  to  detennine  any  range  and 
the  associated  midpoint.  In  this  method  the  proposed  systems  of  the  offerors  are  given 
numerical  ratings,  not  color  or  adjectival  ratings,  between  0  and  100  for  each  factor  and 
subfactor,  and  after  calculating  the  weighted  ratings  using  assigned  weights,  the  sum  of 
the  weighted  numerical  ratings  provides  the  overall  rating  for  each  offeror. 

Hence,  the  steps  that  compose  the  quantitative  weighted  sum  method  are  as 
follows: 

1 .  Determine  factors  and  subfactors 

2.  Determine  relative  importance  of  factors  and  subfactors 

3.  Evaluate  the  proposed  systems  and  give  numerical  ratings  to  factors  and 
subfactors 

4.  Assign  weights  to  factors  and  sub  factors  based  on  their  relative 
importance 

5.  Calculate  weighted  ratings 

6.  Compare  both  offeror's  final  ratings  and  decide  the  winner 

The  important  point  to  emphasize  about  this  model  is  the  fact  that  any  number 
between  0  and  100  can  be  assigned  to  any  factor  or  sub  factor.  Therefore,  when  it  is 
considered  that  in  the  color  rating  method,  only  a  few  numerical  values  can  be  assigned 
to  each  factor  or  subfactor,  the  quantitative  weighted  sum  method  is  more  sensitive  to 
small  differences  between  offerors.  For  instance,  although  two  offerors  that  propose 
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products  with  little  differences  and  one  of  which  is  better  than  the  other  get  most  likely 
the  same  color  ratings,  when  the  quantitative  weighted  sum  method  is  used,  the 
difference  between  offerors  will  be  noticeable  because  of  dissimilar  assigned  numerical 
ratings. 

In  this  model  building  phase,  the  authors  have  once  again  taken  advantage  of 
Microsoft  Excel  and  its  user  friendly  spreadsheets.  One  example  is  going  to  be  used  to 
explain  how  Excel  was  used  in  this  phase  of  the  authors’  study. 

For  the  proposal  risk  factor,  the  appearance  of  the  spreadsheet  before  any  action  is 
shown  in  Figure  14. 
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Figure  14.  The  Appearance  of  the  Weighted  Sum  Method  Spreadsheet  for  the  Proposal 

Risk  Factor  before  Any  Action  in  Version  One. 

For  the  proposal  risk  subfactor  key  system  requirements,  after  evaluation  of  the 
proposed  system,  the  user  enters  an  assessed  numerical  rating  into  cell  C4.  As  soon  as  the 
user  enters  the  rating  into  cell  C4,  Excel  computes  the  weighted  rating  by  multiplying 
that  rating  with  the  related  weight  and  assigns  that  weighted  rating  into  cell  E4  (refer  to 
Figure  15  below). 
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Figure  15.  The  Appearance  of  the  Weighted  Sum  Method  Spreadsheet  for  the  Proposal 
Risk  Factor  after  Entering  the  First  Rating  in  Version  One. 


After  entering  all  the  assessed  numerical  ratings  into  cells  C4-C7,  Excel  calculates 
all  weighted  ratings  and  assigns  them  into  cells  E4-E7.  As  a  final  step,  Excel 
instantaneously  adds  all  weighted  ratings  and  the  overall  numerical  rating  for  the 
proposal  risk  factor,  which  is  58.95  in  this  example,  is  determined  (see  Figure  16). 
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Figure  16.  The  Appearance  of  the  Weighted  Sum  Method  Spreadsheet  for  the  Proposal 
Risk  Factor  after  Entering  All  Ratings  in  Version  One. 


An  example  snapshot  of  the  version  one  weighted  sum  method  spreadsheet  as  a 
whole  is  presented  in  the  Appendix  part  of  the  study. 


C.  MODEL  BUILDING  OF  VERSION  TWO 


In  this  version  of  the  authors’  model,  the  authors  have  made  some  changes  on  the 
color  and  adjectival  ratings  as  well  as  the  ranges  that  are  used  to  determine  the  numerical 
midpoints  for  the  ratings. 
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First  of  all,  the  authors  have  eliminated  some  color  or  adjectival  ratings  that 
reflect  the  lowest  values  for  factors  or  subfactors.  For  instance,  the  red  color  rating  for  the 
mission  capability  subfactors  has  been  discarded  because  the  authors  thought  it  is 
reasonable  to  assume  that  an  offeror  who  gets  a  red  for  any  mission  capability  subfactor, 
which  is  one  of  the  most  critical  requirement  of  a  program,  does  not  deserve  to  compete 
any  more  for  the  contract  because  of  its  unacceptable  offer.  As  another  example,  a  high 
adjectival  rating  for  the  proposal  risk  factor  and  cost  risk  subfactor  has  also  been 
eliminated  to  disqualify  any  offeror  who  puts  high  risk  on  the  program’s  goals. 

In  addition  to  eliminating  some  ratings,  the  authors  have  extended  or  narrowed 
the  numerical  ranges  to  which  color  and  adjectival  ratings  are  related.  The  logic  behind 
this  is  actually  the  desire  to  value  an  offeror  who  proposes  to  provide  more  valuable 
product  to  the  government.  For  instance,  narrowing  the  numerical  range  of  blue,  which  is 
the  highest  color  rating  for  the  mission  capability  subfactors,  increases  the  likelihood  an 
offeror  who  really  deserves  and  gets  that  rating  will  win  the  contract.  On  the  other  hand, 
extending  the  numerical  range  of  green  through  high  points  makes  it  difficult  to  get  a 
blue  rating. 

The  following  diagrams  depicted  in  Figure  17  show  the  new  ranges  and  the 
midpoints  for  the  factors  and  subfactors. 
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Figure  17.  The  Ranges  and  the  Midpoints  of  the  Factors  and  the  Subfactors  for  Version 

Two. 


As  a  last  point,  in  this  version  of  the  authors’  study,  the  IFARA  range  is 
considered  between  1.00  and  2.00  to  disqualify  any  offeror  whose  proposed  system  gives 
a  fleet  effectiveness  value  of  less  than  one  because  it  has  been  assumed  that  a  tanker 
aircraft  which  is  not  even  as  effective  as  the  KC-135  can  not  meet  government  best  value 
requirements.  The  remainder  of  each  model  is  completely  as  same  as  the  initial  ones. 
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VI.  ANALYSIS  AND  RESULTS 


A.  INTRODUCTION 

In  the  KC-X  Tanker  Replacement  Program,  which  is  the  most  significant 
procurement  program  in  the  Air  Force  recently,  the  Air  Force  has  used  its  current  source 
selection  method,  color  rating,  and  the  Airbus/Northrop  Grumman  team  won  the  contract. 
After  debriefings  presented  to  both  offerors,  the  fact  has  emerged  that  the  ratings  of  both 
offerors  are  almost  same  and  they  are  very  similar  where  they  are  not  exactly  the  same. 
Consequently,  long  debates  have  begun  between  Boeing  and  the  Air  Force  and  Boeing 
has  filed  a  protest  with  the  Government  Accountability  Office  (GAO)  asking  the  agency 
to  review  the  award  decision. 

What  if  the  Air  Force  had  used  a  numerical  method  to  determine  the  winner,  not 
the  color  rating  method?  Could  Boeing  have  won  the  contract  and  be  the  USAF’s  choice 
to  build  the  nation’s  next  generation  tanker  plane? 

This  is  the  question  that  the  authors  are  going  to  answer  in  this  chapter  based  on 
the  models  explained  previously.  In  this  chapter,  two  versions  of  the  authors’  models  will 
be  presented  and  it  will  be  demonstrated  on  two  versions  that  if  the  Air  Force  had  used 
the  weighted  sum  method  to  determine  the  winner,  Boeing  might  have  won  the  contract. 
This  result  is  based  on  hypothetical  data  that  would  have  resulted  in  the  actual 
color/adjectival  ratings  given  in  Figure  4.  This  result  will  prove  that  Air  Force’s  source 
selection  process  using  the  color  rating  method  has  some  subjectivity  and  using  a 
numerical  method  might  give  completely  different  results,  thus  leading  another  offeror  to 
end  up  as  the  winner. 

One  important  point  to  state  is  that  while  assigning  numerical  points  to  each 
evaluation  subfactor  in  the  quantitative  weighted  sum  method,  the  authors  have  assigned 
values  among  the  numbers  falling  in  the  range  of  the  associated  color  or  adjectival  rating 
to  demonstrate  that  Boeing  could  have  had  the  better  offer.  For  instance,  in  version  one, 
the  mission  capability  assessment  for  Boeing  when  using  the  quantitative  weighted  sum 
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method;  the  authors  have  assigned  93  (which  is  a  pretty  high  rating)  to  the  key  system 
requirements  subfactor.  They  did  so  because  they  had  to  select  a  number  that  would  be 
able  to  represent  the  blue  rating  and,  therefore,  a  number  between  75  and  100,  which  is 
the  numerical  range  for  the  blue  color  rating  in  version  one. 

B.  THE  ANALYSIS  OF  VERSION  ONE 

Version  one  of  the  authors’  model  has  been  built  based  on  a  basic  logic  in  which 
the  ranges  of  color  and  adjectival  ratings  are  evenly  distributed  and  all  color/adjectival 
ratings  are  taken  into  account  during  the  evaluations.  Version  two,  which  will  be 
analyzed  later  in  the  chapter,  excludes  some  of  the  color/adjectival  ratings. 

In  this  part,  first  of  all,  each  evaluation  factor  will  be  analyzed  separately  for  both 
offerors  using  both  methods  and  overall  factor  ratings  will  be  obtained.  After  that,  overall 
offeror  ratings  will  be  calculated  and  the  differences  between  those  overall  ratings  will  be 
examined. 

There  are  four  tables  in  each  factor  assessment  part  below.  The  first  tables 
demonstrate  Boeing’s  color  ratings  assigned  to  each  subfactor  under  each  factor  for  the 
color  rating  method,  and  the  second  tables  show  its  numerical  ratings  assigned  to  the 
same  subfactors  for  the  quantitative  weighted  sum  method  with  consideration  given  to  the 
numerical  ranges  of  the  color  ratings  assigned  in  the  color  rating  method. 

On  the  other  hand,  the  third  tables  demonstrate  Airbus’s  color  ratings  assigned  to 
each  subfactor  under  each  factor  for  the  color  rating  method,  and  the  fourth  ones  show  its 
numerical  ratings  assigned  to  the  same  subfactors  for  the  quantitative  weighted  sum 
method  considering  the  numerical  ranges  of  the  color  ratings  assigned  in  the  color  rating 
method.  Besides,  overall  factor  ratings  for  each  offeror  calculated  using  both  methods  are 
shown  below  in  tables. 

As  explained  before  in  Chapter  V,  in  models  using  the  color  rating  method  the 
authors  have  converted  the  color  ratings  into  numerical  ratings  using  range  midpoints  in 
order  to  be  able  to  detennine  an  overall  score  to  compare  the  color  rating  method  and  the 
weighted  sum  method.  In  the  models  using  the  weighted  sum  method,  the  numerical 
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values  for  each  factor  and  subfactor  were  selected  by  the  authors,  because  true  values 
were  not  available,  to  demonstrate  how  other  values  within  the  same  color  ranges  could 
yield  different  overall  results.  Also,  all  weights  used  to  show  importance  levels  of  factors 
and  subfactors  were  selected  by  the  authors  with  consideration  given  to  the  importance 
level  requirements  in  the  RFP. 

1.  Mission  Capability  Factor  Assessment 


The  tables  demonstrating  both  offerors’  ratings  obtained  by  using  both  methods 
for  the  mission  capability  factor  are  provided  below  in  Tables  7  and  8. 


Using  Color  Rating  (Table-1)  Using  WSM  Rating  (Table-2) 


MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

93 

0.30 

27.90 

SIS 

64 

0.25 

16.00 

PS 

92 

0.20 

18.40 

PM 

63 

0.15 

9.45 

TMD 

94 

0.10 

9.40 

MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

BLUE 

0.30 

87.5 

26.250 

SIS 

GREEN 

0.25 

62.5 

15.625 

j  PS 

BLUE 

0.20 

87.5 

17.500 

PM 

GREEN 

0.15 

62.5 

9.375 

TMD 

GREEN 

0.10 

83.3 

8.330 

77.080 

Table  7.  Mission  Capability  Factor  Assessment  for  Boeing. 


Using  Color  Rating  (Table-3)  Using  WSM  Rating  (Table-4) 


MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

BLUE 

0.30 

87.5 

26.250 

KSR 

81 

0.30 

24.30 

SIS 

GREEN 

0.25 

62.5 

15.625 

SIS 

60 

0.25 

15.00 

PS 

BLUE 

0.20 

87.5 

17.500 

PS 

90 

0.20 

18.00 

PM 

GREEN 

0.15 

62.5 

9.375 

PM 

58 

0.15 

8.70 

TMD 

GREEN 

0.10 

83.3 

8.330 

TMD 

85 

0.10 

8.50 

77.080 

Table  8.  Mission  Capability  Factor  Assessment  for  Airbus. 


As  seen  on  the  tables  above,  while  Boeing  and  Airbus  both  get  77.08  as  their 
overall  mission  capability  factor  rating  when  the  color  rating  method  is  used  because  they 
have  the  same  color  ratings  for  these  subfactors,  Boeing  gets  81.15  and  Airbus  gets  74.50 
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when  the  numerical  method  is  used.  This  result  demonstrates  that  although  both  offerors 
received  the  same  color  ratings,  Boeing  could  possibly  have  proposed  a  technically  better 
product  to  the  government. 


2.  Proposal  Risk  Factor  Assessment 


Tables  9  and  10  show  proposal  risk  assessment  results  of  both  offerors  obtained 
via  both  methods. 


Using  Color  Rating  (Table-1) 


Using  WSM  Rating  (Table-2) 


PROPOSAL  RISK 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

LOW 

0.35 

83.3 

29.155 

SIS 

MODERATE 

0.30 

50.0 

15.000 

PS 

LOW 

0.20 

83.3 

16.660 

PM 

LOW 

0.15 

83.3 

12.495 

PROPOSAL  RISK 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

94 

0.35 

32.90 

SIS 

57 

0.30 

17.10 

PS 

97 

0.20 

19.40 

PM 

94 

0.15 

14.10 

83.50 

Table  9.  Proposal  Risk  Factor  Assessment  for  Boeing. 


Using  Color  Rating  (Table-3)  Using  WSM  Ratin§  (Table-4) 


PROPOSAL  RISK 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

LOW 

0.35 

83.3 

29.155 

SIS 

MODERATE 

0.30 

50.0 

15.000 

PS 

LOW 

0.20 

83.3 

16.660 

PM 

LOW 

0.15 

83.3 

12.495 

PROPOSAL  RISK 

WGHT 

WEIGHTD 

SUBFACTORS 

RATINGS 

KSR 

85 

0.35 

29.75 

SIS 

57 

0.30 

17.10 

PS 

91 

0.20 

18.20 

PM 

87 

0.15 

13.05 

Table  10.  Proposal  Risk  Factor  Assessment  for  Airbus. 


For  the  proposal  risk  factor,  because  both  offerors  have  received  the  same 
adjectival  ratings,  they  get  73.31  as  their  overall  proposal  risk  factor  rating.  However, 
when  the  quantitative  weighted  sum  method  is  used,  while  Airbus  gets  78.10,  Boeing  gets 
a  higher  score  of  83.50.  This  result  demonstrates  that  although  both  offerors  received  the 
same  color  ratings,  Boeing  could  have  proposed  a  product  to  the  government  which  is 
less  risky  for  the  program’s  objectives. 
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3. 


Past  Performance  Assessment 


Both  offerors’  ratings  obtained  by  using  both  methods  for  the  past  performance 
factor  are  provided  below  in  Tables  1 1  and  12. 


Using  Color  Rating  (Table-1)  Using  WSM  Rating  (Tabie_2) 


Table  1 1 .  Past  Perfonnance  Assessment  for  Boeing. 


Using  Color  Rating  (Table-3) 


PAST  PERFORMANCE 

RATING 

SATISFACTORY 

58.35 

58.35 


Using  WSM  Rating  (Table-4) 


PAST  PERFORMANCE 

RATING 

56 

56 

Table  12.  Past  Performance  Assessment  for  Airbus. 


For  the  past  performance  factor,  both  offerors  have  received  the  rating  of 
‘satisfactory’  which  results  in  the  same  score  for  both  offerors  when  the  color  rating 
method  is  used.  However,  Boeing  gets  a  higher  point  value  when  the  numerical  method  is 
used  because  of  the  assumption  that  its  past  performance  is  better  than  Airbus’s. 

4.  Cost/Price  Factor  Assessment 

Below  are  the  tables  showing  the  cost/price  factor  assessment  results  obtained 
using  both  methods  of  both  offerors  (Tables  13  and  14). 

Using  Color  Rating  (Table-1)  Using  WSM  Ratin§  (Table-2) 


COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.044 

0.60 

39.12 

23.472 

RISK 

MODERATE 

0.40 

50.00 

20.000 

43.472 

COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.044 

0.60 

39.12 

23.472 

RISK 

58 

0.40 

58.00 

23.200 

46.672 

Table  13.  Cost/Price  Factor  Assessment  for  Boeing 
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Using  Color  Rating  (Table-3) 


Using  WSM  Rating  (Table-4) 


COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.01 

0.60 

39.8 

23.88 

RISK 

LOW 

0.40 

83.3 

33.32 

57.20 

COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.01 

0.60 

39.80 

23.88 

RISK 

70 

0.40 

70.00 

28.00 

Table  14.  Cost/Price  Factor  Assessment  for  Airbus. 


The  tables  above  show  that  when  the  color  rating  method  is  used,  Airbus  gets  a 
considerably  higher  score  than  Boeing  in  regards  to  the  cost/price  factor.  However,  when 
the  quantitative  weighted  sum  method  is  used,  the  difference  between  each  offeror’s 
cost/price  factor  ratings  substantially  shrinks  because  of  selecting  a  high  point  (58.00) 
among  points  falling  in  the  ‘moderate’  risk  rating  numerical  range  (i.e.,  33-66)  and 
assigning  it  as  the  Boeing’s  numerical  rating  for  the  cost  risk  subfactor.  The  assumption 
for  assigning  a  high  point  value  representing  the  ‘moderate’  risk  rating  to  Boeing  is  that  it 
might  have  proposed  a  life  cycle  cost  that  put  some  risk  on  the  program’s  goals  and  that 
risk  was  slightly  more  than  what  is  allowed  in  order  to  get  a  Tow’  risk  rating. 


5.  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA) 


Below  in  Tables  15  and  16  are  the  evaluations  of  the  IFARA  factor  for  both 
offerors  using  both  methods.  Since  there  is  not  any  subfactor  under  the  IFARA  factor  and 
no  color  or  adjectival  ratings  have  been  used  in  this  factor  evaluation,  no  difference  can 
be  observed  between  each  offeror’s  ratings.  Airbus  gets  a  higher  point  value  than  Boeing 
because  of  its  higher  fleet  effectiveness  value  using  both  methods. 

Using  Color  Rating  Using  Numerical  Rating 


IFARA 

RATING 

1.79 

83.2 

83.2 

IFARA 

RATING 

1.79 

83.2 

83.2 

Table  15.  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA)  for  Boeing. 
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Using  Color  Rating 


Using  Numerical  Rating 


IFARA 

RATING 

1.9 

92 

92 

IFARA 

RATING 

1.9 

92 

92 

Table  16.  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA)  for  Airbus. 


6.  Conclusion 


The  tables  presenting  all  weighted  subfactor  ratings  and  calculated  overall  offeror 
ratings  for  both  methods  and  offerors  are  provided  below  in  Tables  17  and  18. 


Using  Color  Rating 


Using  WSM  Rating 


FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

77.08 

19.270 

Proposal  Risk 

0.250 

73.31 

18.328 

Past  Performance 

0.250 

58.35 

14.588 

Cost/Price 

0.125 

43.47 

5.434 

IFARA 

0.125 

83.20 

10.400 

OVERALL  OLLEROR  RATING: 

68.019 

LACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

81.15 

20.288 

Proposal  Risk 

0.250 

83.50 

20.875 

Past  Performance 

0.250 

59.00 

14.750 

Cost/Price 

0.125 

46.67 

5.834 

IFARA 

0.125 

83.20 

10.400 

OVERALL  OFFEROR  RATING: 

72.147 

Table  17.  Overall  Scores  for  Boeing. 


Using  Color  Rating 


Using  WSM  Rating 


FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

77.08 

19.270 

Proposal  Risk 

0.250 

73.31 

18.328 

Past  Performance 

0.250 

58.35 

14.588 

Cost/Price 

0.125 

57.20 

7.150 

IFARA 

0.125 

92.00 

11.500 

OVERALL  OFFEROR  RATING: 

70.835 

FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

74.50 

18.625 

Proposal  Risk 

0.250 

78.10 

19.525 

Past  Performance 

0.250 

56.00 

14.000 

Cost/Price 

0.125 

51.88 

6.485 

IFARA 

0.125 

92.00 

11.500 

OVERALL  OFFEROR  RATING: 

70.135 

Table  18.  Overall  Scores  for  Airbus. 


The  results  are  really  noteworthy  considering  the  importance  of  the  KC-X  Tanker 
Replacement  Program.  As  easily  seen  above,  there  are  two  different  winners  to  whom  the 
contract  should  be  awarded  when  using  two  kinds  of  methods.  When  using  the  color 
rating  method.  Airbus  gets  a  score  of  70.835  and  Boeing  gets  a  score  of  68.019,  which 
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means  Airbus  deserves  to  get  the  contract.  On  the  other  hand,  when  the  quantitative 
weighted  sum  method  is  used,  Airbus  gets  an  overall  score  of  70.135  and  Boeing  gets 
72. 147,  so  the  winner  should  be  Boeing. 

The  implication  under  these  results  is  that  when  there  are  little  but  important 
differences  between  offerors,  the  color  rating  method  is  potentially  incompetent  in 
reflecting  those  differences.  However,  the  quantitative  weighted  sum  method  is  capable 
of  handling  those  little  details  and  providing  more  accurate  results. 

The  snapshots  of  all  the  models  and  a  comparison  chart  for  version  one  can  be 
found  in  the  Appendix  part  of  the  study. 

C.  THE  ANALYSIS  OF  VERSION  TWO 

As  explained  in  the  previous  chapter,  version  two  differs  from  version  one  based 
on  some  changes  of  the  color/adjectival  ratings  and  the  ratings’  numerical  ranges. 
Initially,  the  color/adjectival  ratings  that  reflect  the  lowest  values  for  factors/subfactors 
are  eliminated  and,  then,  the  numerical  ranges  to  which  the  color  and  adjectival  ratings 
are  related  to  are  extended  or  narrowed.  For  instance,  the  high  adjectival  rating  for  the 
proposal  risk  factor  and  cost  risk  subfactor  has  been  eliminated  to  disqualify  any  offeror 
who  puts  high  risk  on  the  program’s  goals,  and  the  range  of  the  low  risk  rating  has  been 
narrowed  to  value  the  offeror  who  proposes  a  less  risky  product  to  the  government.  More 
detailed  explanations  about  the  logic  behind  these  changes  were  provided  in  the  previous 
chapter. 

In  this  part,  first  of  all,  each  evaluation  factor  will  be  analyzed  separately  for  both 
offerors  using  both  methods.  After  overall  factor  ratings  are  obtained,  overall  offeror 
ratings  will  be  calculated  and  the  differences  between  those  overall  ratings  will  be 
discussed. 
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There  are  four  tables  in  each  factor  assessment  part  below.  The  first  tables 
demonstrate  Boeing’s  color  ratings  assigned  to  each  subfactor  under  each  factor  for  the 
color  rating  method,  and  the  second  tables  show  its  numerical  ratings  assigned  to  the 
same  subfactors  for  the  quantitative  weighted  sum  method  with  consideration  given  to  the 
numerical  ranges  of  the  color  ratings  assigned  in  the  color  rating  method. 

On  the  other  hand,  the  third  tables  demonstrate  Airbus’s  color  ratings  assigned  to 
each  subfactor  under  each  factor  for  the  color  rating  method,  and  the  fourth  ones  show  its 
numerical  ratings  assigned  to  the  same  subfactors  for  the  quantitative  weighted  sum 
method  considering  the  numerical  ranges  of  the  color  ratings  assigned  in  the  color  rating 
method.  Besides,  overall  factor  ratings  for  each  offeror  calculated  using  both  methods  are 
shown  below  the  tables. 

At  this  point,  the  authors  want  to  remind  the  reader  that  the  difference  between 
the  analysis  tables  for  versions  one  and  two  is  that  in  version  two  the  color  rating  method 
rating  values  reflect  new  midpoints  for  the  color  ratings. 

1.  Mission  Capability  Factor  Assessment 

The  tables  showing  the  mission  capability  factor  assessment  results  of  both 
offerors  that  were  obtained  using  both  methods  are  provided  below  in  Tables  19  and  20. 

Using  Color  Rating  (Table-1)  Using  WSM  Rating  (Table-2) 


MISSION  CAPABILITY 

WGHT 

WEIGHTD 

SUBFACTORS 

RATINGS 

KSR 

93 

0.30 

27.90 

SIS 

64 

0.25 

16.00 

PS 

92 

0.20 

18.40 

PM 

63 

0.15 

9.45 

TMD 

94 

0.10 

9.40 

81.15 

MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

BLUE 

0.30 

82.50 

24.75 

j  SIS 

GREEN 

0.25 

45.00 

11.25 

PS 

BLUE 

0.20 

82.50 

16.50 

PM 

GREEN 

0.15 

45.00 

6.75 

TMD 

GREEN 

0.10 

80.00 

8.00 

67.25 

Table  19.  Mission  Capability  Factor  Assessment  for  Boeing. 
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Using  Color  Rating  (Table-3) 


Using  WSM  Rating  (Table-4) 


MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

81 

0.30 

24.30 

SIS 

60 

0.25 

15.00 

PS 

90 

0.20 

18.00 

PM 

58 

0.15 

8.70 

TMD 

85 

0.10 

8.50 

MISSION  CAPABILITY 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

BLUE 

0.30 

82.50 

24.75 

SIS 

GREEN 

0.25 

45.00 

11.25 

PS 

BLUE 

0.20 

82.50 

16.50 

PM 

GREEN 

0.15 

45.00 

6.75 

TMD 

GREEN 

0.10 

80.00 

8.00 

67.25 

Table  20.  Mission  Capability  Factor  Assessment  for  Airbus. 


Since  Boeing  and  Airbus  have  the  same  color  ratings  for  the  mission  capability 
subfactors,  they  both  get  67.25  as  their  overall  factor  rating  when  the  color  rating  method 
is  used.  However,  when  the  quantitative  weighted  sum  method  is  used,  Boeing  gets  a 
score  of  81.15  and  Airbus  gets  74.50.  The  answer  to  how  Boeing  may  get  a  higher  score 
when  the  numerical  method  is  used  is  hidden  behind  the  features  of  the  quantitative 
weighted  sum  method.  Due  to  the  fact  that  it  is  more  capable  of  showing  little  differences 
between  offerors,  the  final  scores  may  be  very  different  although  they  may  be  exactly  the 
same  when  the  color  rating  method  is  used.  In  this  particular  case,  Boeing  might  have 
proposed  a  more  technically  capable  product  to  the  government. 

2.  Proposal  Risk  Factor  Assessment 


The  proposal  risk  factor  assessment  tables  that  demonstrate  all  scores  for  both 
offerors  and  methods  are  shown  below  in  Tables  21  and  22. 


Using  Color  Rating  (Table-1) 


Using  WSM  Rating  (Table-2) 


PROPOSAL  RISK 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

LOW 

0.35 

80.00 

28.00 

SIS 

MODERATE 

0.30 

30.00 

9.00 

PS 

LOW 

0.20 

80.00 

16.00 

PM 

LOW 

0.15 

80.00 

12.00 

PROPOSAL  RISK 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

94 

0.35 

32.90 

SIS 

57 

0.30 

17.10 

PS 

97 

0.20 

19.40 

PM 

94 

0.15 

14.10 

83.50 

Table  2 1 .  Proposal  Risk  Factor  Assessment  for  Boeing. 
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Using  Color  Rating  (Table-3) 


Using  WSM  Rating  (Table-4) 


PROPOSAL  RISK 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

KSR 

LOW 

0.35 

80.00 

28.00 

SIS 

MODERATE 

0.30 

30.00 

9.00 

PS 

LOW 

0.20 

80.00 

16.00 

PM 

LOW 

0.15 

80.00 

12.00 

PROPOSAL  RISK 

SUBFACTORS 

WGHT 

WEIGHTD 

RATINGS 

KSR 

85 

0.35 

29.75 

SIS 

57 

0.30 

17.10 

PS 

91 

0.20 

18.20 

PM 

87 

0.15 

13.05 

78.10 

Table  22.  Proposal  Risk  Factor  Assessment  for  Airbus. 


These  results  are  very  similar  to  the  previous  ones.  Although  both  offerors  get  the 
very  same  score,  which  is  65,  for  the  proposal  risk  factor  when  the  color  rating  method  is 
used,  Boeing’s  score  is  higher  than  the  Airbus  score  when  the  quantitative  weighted  sum 
method  is  used.  The  possible  reason  for  this  is  that  Boeing  proposed  a  product  that  puts 
lower  risk  on  the  schedule,  cost  and  performance  goals  even  though  they  received  the 
same  adjectival  ratings. 


3.  Past  Performance  Assessment 


The  tables  of  the  past  performance  assessment  for  both  offerors  using  both 
methods  are  given  below  in  Tables  23  and  24. 

Using  Color  Rating  (Table-1)  Using  WSM  Rating  (Table-2) 


Table  23.  Past  Perfonnance  Assessment  for  Boeing. 


PAST  PERFORMANCE 

RATING 

59 

59 

PAST  PERFORMANCE 

RATING 

SATISFACTORY 

45 

45 


Using  Color  Rating  (Table-3)  Using  WSM  Rating  (Table-4) 


Table  24.  Past  Performance  Assessment  for  Airbus. 
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For  the  past  performance  factor,  both  offerors  have  received  the  rating  of 
‘satisfactory’  that  causes  the  same  score  for  both  offerors  when  the  color  rating  method  is 
used.  However,  Boeing  gets  a  higher  point  value  when  the  numerical  method  is  used 
because  of  the  assumption  that  its  past  perfonnance  is  better  than  Airbus  and  almost  high 
enough  to  get  a  ‘significant’  past  performance  rating. 

4.  Cost/Price  Factor  Assessment 


Tables  25  and  26  show  the  cost/price  factor  assessment  results  of  both  offerors 
that  were  obtained  using  both  methods. 


Using  Color  Rating  (Table-1) 


Using  WSM  Rating  (Table-2) 


COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.044 

0.60 

39.12 

23.472 

RISK 

MODERATE 

0.40 

30.00 

12.000 

35.472 

COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.044 

0.60 

39.12 

23.472 

RISK 

58 

0.40 

58.00 

23.200 

46.672 

Table  25.  Cost/Price  Factor  Assessment  for  Boeing. 

Using  Color  Rating  (Table-3)  Using  WSM  Rating  (Table-4) 


COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.01 

0.60 

39.80 

23.88 

RISK 

LOW 

0.40 

80.00 

32.00 

55.88 

COST/PRICE 

SUBFACTORS 

WGHT 

RATNG 

WEIGHTD 

RATINGS 

COST 

108.01 

0.60 

39.80 

23.88 

RISK 

70 

0.40 

70.00 

28.00 

Table  26.  Cost/Price  Factor  Assessment  for  Airbus. 


For  the  cost/price  factor,  Airbus  always  gets  a  relatively  higher  score  than  Boeing 
when  either  method  is  used  because  of  its  Tow’  cost  risk  rating.  However,  as  easily  seen 
on  the  tables  above,  the  difference  between  the  overall  factor  ratings  of  the  offerors 
diminish  when  the  quantitative  weighted  sum  method  is  used.  The  assumption  that  leads 
the  authors  to  reach  this  result  is  that  while  Boeing  might  have  proposed  a  life  cycle  cost 
that  is  not  realistic  enough  to  get  a  Tow’  risk  rating  but  very  close  to  it,  Airbus  might 
have  proposed  a  cost  that  is  slightly  more  realistic  than  Boeing  and  is  able  get  a  Tow’  risk 
rating. 
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5. 


Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA) 


The  evaluations  of  the  IFARA  factor  for  both  offerors  obtained  using  both 
methods  are  given  below  in  Tables  27  and  28. 


Using  Color  Rating  (Table-1) 


Using  WSM  Rating  (Table-2) 


IFARA 

RATING 

1.79 

79 

79 

IFARA 

RATING 

1.79 

79 

79 

Table  27.  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA)  for  Boeing. 


Using  Color  Rating  (Table-3) 


IFARA 

RATING 

1.9 

90 

Using  WSM  Rating  (Table-4) 


IFARA 

RATING 

1.9 

90 

Table  28.  Integrated  Fleet  Aerial  Refueling  Assessment  (IFARA)  for  Airbus. 


Since  the  IFARA  assessment  does  not  include  any  color/adjectival  rating,  there  is 
no  difference  between  each  offeror’s  ratings.  Airbus  gets  a  higher  point  score  than 
Boeing  because  of  its  higher  fleet  effectiveness  value  using  both  methods. 


6,  Conclusion 


The  tables  presenting  all  weighted  subfactor  ratings  and  calculated  overall  offeror 
ratings  for  both  methods  and  offerors  are  provided  below  in  Tables  29  and  30. 


Using  Color  Rating 


Using  WSM  Rating 


FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

67.250 

16.813 

Proposal  Risk 

0.250 

65.000 

16.250 

Past  Performance 

0.250 

45.000 

11.250 

Cost/Price 

0.125 

35.472 

4.434 

IFARA 

0.125 

79.000 

9.875 

OVERALL  OLLEROR  RATING: 

58.622 

LACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

81.150 

20.288 

Proposal  Risk 

0.250 

83.500 

20.875 

Past  Performance 

0.250 

59.000 

14.750 

Cost/Price 

0.125 

46.672 

5.834 

IFARA 

0.125 

79.000 

9.875 

OVERALL  OFFEROR  RATING: 

71.622 

Table  29.  Overall  Scores  for  Boeing. 
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Using  Color  Rating 


Using  WSM  Rating 


FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.25 

67.25 

16.813 

Proposal  Risk 

0.25 

65.00 

16.250 

Past  Performance 

0.25 

45.00 

11.250 

Cost/Price 

0.125 

55.88 

6.985 

IFARA 

0.125 

90.00 

11.250 

62.548 

FACTORS 

WGHT 

RATNG 

WEIGHTD 

RATING 

Mission  Capability 

0.250 

74.50 

18.625 

Proposal  Risk 

0.250 

78.10 

19.525 

Past  Performance 

0.250 

56.00 

14.000 

Cost/Price 

0.125 

51.88 

6.485 

IFARA 

0.125 

90.00 

11.250 

69.885 

Table  30.  Overall  Scores  for  Airbus. 


Version  two  of  the  authors’  model,  which  suggests  a  different  way  to  evaluate 
both  offerors,  results  in  the  same  significant  result  as  was  obtained  in  the  first  version.  In 
this  version,  although  Airbus  gets  the  higher  overall  score,  which  is  62.548,  when  the 
color  rating  method  is  used,  it  gets  a  lower  overall  score,  which  is  69.885,  when  the 
quantitative  weighted  sum  method  is  used.  Therefore,  there  is  a  change  in  the  winner  and 
Boeing  gets  the  contract  when  the  numerical  method  is  used. 

This  part  of  the  authors’  study  supports  their  argument  that  the  quantitative 
weighted  sum  method  is  more  competent  than  the  color  rating  method  in  reflecting  the 
little  but  important  differences  between  offerors. 

The  snapshots  of  all  the  models  and  a  comparison  chart  for  version  two  can  be 
found  in  the  Appendix  part  of  the  study. 

In  the  following  chapter,  three  different  factors  will  be  evaluated  to  understand 
their  effects  on  the  results  via  sensitivity  analysis.  Sensitivity  analysis  will  be  performed 
to  demonstrate  how  these  results  are  impacted  by  changing  the  choice  of  weights.  In 
addition,  how  susceptible  these  results  are  to  the  selection  of  the  rating  values  with  the 
color  ranges  will  be  explored. 
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VII.  SENSITIVITY  ANALYSIS 


A.  INTRODUCTION 

In  this  chapter,  the  authors  will  investigate  how  modeling  choices  might  affect  the 
results  that  were  presented  in  the  previous  chapter.  In  the  previous  chapter,  the  authors’ 
study  results  support  their  argument  that  the  quantitative  weighted  sum  method  is  more 
competent  than  the  color  rating  method  in  reflecting  the  little  but  important  differences 
between  offerors.  The  color  rating  method  is  potentially  incompetent  in  reflecting  those 
differences. 

In  this  part  of  the  study,  sensitivity  analysis  will  be  done  to  understand  the  effects 
and  limits  of  ratings  (within  color  ranges)  and  weights  in  the  models.  Three  different 
forms  of  sensitivity  analysis  will  be  done.  In  the  first  one,  numerical  ratings  for  the  color 
and  adjectival  ratings  will  be  changed  within  their  ranges  to  see  the  best/worst  possible 
overall  scores  for  each  alternative  in  version  one  of  the  authors’  model.  In  the  second 
one,  the  same  analysis  that  will  be  done  in  the  first  sensitivity  analysis  will  be  applied  to 
the  authors’  version  two  model  (with  a  diminished  number  of  color  and  adjectival 
ratings).  In  the  third  final  one,  the  weights  will  be  changed.  While  keeping  original  model 
ratings  as  constant,  the  authors  will  analyze  the  effects  of  weights  on  results.  The  last 
analysis  will  be  done  for  each  alternative  and  for  the  color  and  weighted  sum  method 
(WSM)  rating  techniques. 

B.  FIRST  SENSITIVITY  ANALYSIS 

In  the  previous  chapter,  the  results  show  that  while  Airbus  wins  the  contract  with 
the  color  rating  method  evaluations,  in  the  WSM  evaluation  Airbus  loses  it  with 
numerical  scores  that  are  in  the  same  color  ranges  (numerical  equivalents  of  color 
ratings).  Based  on  these  results,  the  authors  decided  to  perform  a  sensitivity  analysis  to 
understand  the  effects  of  the  color  rating  ranges  on  the  results.  The  authors  will  keep  the 
color  rating  ranges  as  constant  and  will  change  the  numerical  ratings  that  are  equivalent 
to  those  color  ratings  in  the  possible  rating  ranges.  The  research  question  is  “How 
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sensitive  is  the  overall  WSM  score  to  different  numerical  ratings  within  each  color 
range?”  This  will  give  the  authors  the  best  and  the  worst  overall  score  that  each  company 
could  get  using  the  numerical  model  of  their  study  (while  keeping  numerical  ratings 
within  color  ranges  for  assigned  colors). 

The  authors  used  the  formula  on  the  following  page  to  do  the  first  part  of  their 
sensitivity  analysis.  To  calculate  the  best  overall  score,  the  authors  needed  a 
maximization  model  for  the  objective  function,  and  to  calculate  the  worst  overall  score, 
they  needed  a  minimization  model  for  the  objective  function.  Each  (max  and  min)  model 
was  implemented  twice  -  one  for  Boeing  and  one  for  Airbus. 

The  formulation  on  the  next  page  is  for  Boeing  in  version  one  of  the  authors’ 
model.  With  some  little  changes,  this  fonnula  adapted  to  Airbus  so  that  the  same  analysis 
could  be  performed.  The  following  model  will  be  solved  once  for  maximization  and  the 
results  will  give  the  best  possible  score  for  Boeing  with  the  assigned  color  and  adjectival 
rating  ranges  in  the  first  version  of  the  authors’  model.  Next,  it  will  be  solved  for 
minimization  and  the  results  will  give  the  worst  possible  score  for  Boeing  with  the 
assigned  color  and  adjectival  rating  ranges  in  the  first  version  of  the  authors’  model. 
Then,  the  adapted  version  of  the  following  fonnula  will  be  solved  two  more  times,  once 
for  maximization  and  once  for  minimization,  to  reach  the  results  for  possible  best/worst 
scenario  overall  scores  for  Airbus. 

In  the  model,  all  the  sub  factors  that  are  assigned  with  color  and  adjectival  ratings 
will  be  the  decision  variables.  The  factors  and  subfactors,  those  get  specific  numerical 
ratings  like  IFARA  and  life  cycle  cost,  will  not  be  included  in  the  model  as  decision 
variables.  The  objective  function  will  be  an  overall  score  function  that  consists  of  all 
subfactor  ratings  and  their  weights,  and  the  constraints  will  be  the  max  and  min  values  for 
each  sub  factor  with  color  and  adjectival  ratings.  For  example,  for  KSR,  decision  variable 
xj  can  get  any  value  between  76  and  100  (including  76  and  100).  Since  all  the  decision 
variables  will  be  in  a  positive  number  range,  there  will  not  be  a  non-negativity  constraint 
(redundant  constraint-this  constraint  does  not  add  any  value  to  the  model).  Also,  the 
model  will  be  solved  as  a  linear  model. 
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MODEL(S)  FOR  BEST/WORST  RATINGS  WITHIN  COLOR  RANGES 
(FOR  BOEING  IN  VERSION  1) 


Decision  Variables: 


xj  :  Mission  Capability  (MC)  subfactor  Key  System  Requirements  (KSR)  rating. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  rating. 

X3  :  MC  subfactor  Product  Support  (PS)  rating. 

X4  :  MC  subfactor  Program  Management  (PM)  rating. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  rating. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  rating. 

X7  :  PR  subfactor  System  Integration  and  Software  (SIS)  rating. 

xs  :  PR  subfactor  Product  Support  (PS)  rating. 

xg  :  PR  subfactor  Program  Management  (PM)  rating. 

xio  :  Cost/Price  subfactor  Cost  Risk  (CR)  rating. 

Xu  :  Past  Performance  (PP)  rating. 

Objective  Function  (Linear  Model): 


Maximize  Overall  Score  =  (0.3  xi  +  0.25  x?  +  0.2  x.?  +0.15  x4  +  0. 1  xs )  0.25  + 

or  (0.35  xg  +  0.3  X7  +  0.20  x«  +  0. 1 5  X9)  0.25  + 

Minimize  0.25  Xu  +  0.125  *  83.2  + 

(0.40  x;o  +  0.60  *  39.12)  0.125 

Constraints: 


For  Mission  Capability; 
xi  >=  76  andx;  <=100 
X2>=  51  and  X2  <=75 
X3  >=  76  andxs  <=100 
x4  >=  5 1  and  x4  <=75 
xj  >=  61  andx5  <=100 

For  Proposal  Risk; 

X6  >=  67  and  xg  <=  1 00 
X7  >=  34  and  X7  <=66 
xs  >=  67  andxs  <=100 
xg  >=  67  and  xg  <=  1 00 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 

(Sub factor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Subfactor  TMD  is  within  green  color  rating  range) 

(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

xio  >=  34  and  xw  <=66  (CR  is  within  moderate  adjectival  rating  range) 

For  Past  Performance; 

xu>=  51  and  Xu  <=66  (PP  is  within  satisfactory  adjectival  rating  range) 
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To  solve  the  model,  Microsoft  Excel  2003 ’s  Solver  tool  was  used  and  a  snapshot 
of  the  spreadsheet  is  presented  in  the  Appendix  section  of  this  study. 

The  model  above  is  adapted  to  Airbus  by  changing  the  IFARA  rating  from  83.2  to 
92.0  and  the  life  cycle  cost  (LCC)  rating  from  39.12  to  39.80  in  the  objective  function  for 
overall  score  (because  these  results  are  evaluated  directly  as  numerical,  they  are  placed  as 
constant  values  in  the  model).  Also,  changing  the  ranges  for  xw  (cost  risk  sub  factor)  is 
necessary,  because  Airbus  received  a  Tow’  adjectival  rating  instead  of  a  ‘moderate’ 
rating  (Boeing’s  CR  rating)  for  this  subfactor.  The  new  range  for  xjo  is  67  to  100  instead 
of  34  to  66.  The  model  formulation  for  Airbus  is  also  presented  in  the  Appendix  section 
of  this  study. 

As  expected  in  the  solution  of  the  models,  while  solving  for  maximization,  all  the 
ratings  (decision  variables)  got  the  highest  possible  points  in  their  range,  and  while 
solving  for  minimization,  all  the  ratings  got  the  lowest  possible  points  in  their  range.  For 
example,  for  mission  capability  subfactor  KSR,  Boeing  originally  received  a  blue  color 
rating  which  ranges  from  76  to  100  on  a  100  scale.  When  solving  for  maximization,  the 
authors’  model  assigned  100  points  as  the  rating  for  this  subfactor,  and  when  solving  for 
minimization,  the  model  assigned  76  points. 

The  overall  scores  for  Boeing  are  78.084  as  the  maximum  and  58.334  as  the 
minimum.  For  Airbus,  the  scores  are  80.935  as  the  maximum  and  61.135  as  the 
minimum.  The  ranges  for  max/min  overall  scores  are  19.75  for  Boeing  and  19.80  for 
Airbus.  Airbus’s  max/min  scores  are  both  almost  2.80  point  higher  than  Boeing’s  scores, 
but  their  scores  are  overlapping  for  16.95  points.  These  results  are  also  presented  in  Table 
31. 


VERSION  1 

BOEING 

AIRBUS 

DIFFERENCE  OF  BOEING/AIRBUS 

MAX 

78.084 

80.935 

2.851 

MIN 

58.334 

61.135 

2.801 

RANGE 

19.75 

19.8 

Table  3 1 .  Summary  Results  Table  for  Sensitivity  Analysis  Version  One. 
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The  sensitivity  of  the  overall  scores  for  both  alternatives  is  very  similar.  This  is 
because  of  the  fact  that  while  using  color  and  adjectival  rating  methods  for  evaluation, 
the  scores  for  the  two  alternatives  are  very  similar.  There  are  just  three  differences  -  one 
in  the  IFARA,  one  in  the  LCC,  and  one  in  the  cost  risk  rating.  Only  three  out  of  the 
thirteen  criteria  that  affect  the  overall  score  are  different  and  these  are  only  slightly 
different.  In  addition,  since  these  three  factors  are  also  multiplied  by  small  weights,  they 
make  even  a  smaller  difference  in  the  overall  score.  Another  important  finding  from  these 
results  is  the  high  percentage  (almost  85%;  16.95/19.80)  of  overlap  between  the  ranges  of 
possible  points  for  both  alternatives.  This  overlapping  area  means  that  it  is  possible  for 
either  alternative  to  win  the  contract  whenever  the  overall  scores  for  both  are  in  this 
overlapping  range. 

C.  SECOND  SENSITIVITY  ANALYSIS 

For  the  second  part  of  the  sensitivity  analysis,  the  authors  will  follow  the  same 
methodology  as  the  sensitivity  analysis  in  the  first  part  and  apply  it  to  the  second  model. 
The  authors  will  examine  the  results  of  the  analysis  to  understand  the  effects  of  color 
rating  ranges  on  the  overall  scores.  They  will  again  keep  the  color  rating  ranges  as 
constant  and  will  change  the  numerical  ratings  that  are  equivalent  of  those  color  ratings  in 
the  possible  rating  ranges.  In  this  part,  because  the  authors  will  be  working  on  their 
second  version,  they  will  not  use  some  of  the  color  and  adjectival  ratings  that  were 
effective  in  the  first  model  (like  red  in  the  color  ratings,  ’high'  in  the  risk  ratings  and  'no 
confidence’  in  the  performance  ratings),  so  the  color  and  adjectival  rating  ranges  will  be 
changed. 

The  research  question  is  again  “How  sensitive  is  the  overall  WSM  score  to 
different  numerical  ratings  within  each  color  range  (with  a  diminished  number  of  color 
and  adjectival  ratings)?”  This  will  again  give  the  authors  the  best  and  the  worst  overall 
score  that  each  company  could  get  using  the  WSM  model  of  their  study  (while  keeping 
numerical  ratings  within  color  ranges  for  assigned  colors).  Then,  the  authors  will  also 
examine  the  effects  of  diminishing  the  effective  number  of  color  and  adjectival  ratings. 
They  will  calculate  the  best  and  worst  overall  score  in  the  same  way,  that  is  a 
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maximization  model  for  the  objective  function  for  the  best  overall  score  and  a 
minimization  model  for  the  objective  function  for  the  worst  overall  score.  Each  (max  and 
min)  model  will  be  implemented  twice,  one  for  Boeing  and  one  for  Airbus. 

The  formulation  for  Boeing  in  the  second  part  of  the  sensitivity  analysis  will  have 
the  same  decision  variables,  a  very  similar  objective  function  (changes  are  highlighted), 
and  very  similar  constraints.  The  only  part  that  will  be  different  from  the  formulation  for 
Boeing  in  version  one  is  the  ranges  that  are  the  numbers  in  the  right-hand  side  of  the 
equations  for  constraints.  The  new  objective  function  and  the  new  constraints  with  the 
new  ranges  as  the  right-hand  side  values  of  the  equations  are  as  follows: 


Objective  Function: 


Maximize 

or 

Minimize 

Constraints: 


Overall  Score  =  (0.3  xj  +  0.25  x?  +  0.2  X3  +  0.15  X4+  0.1  xj)  0.25  + 
(0.35  X6  +  0.3  X7  +  0.20  xs  +  0.15  xg)  0.25  + 

0.25  xn  +  0.125  *  79  + 

(0.40  xio  +  0.60  *  39.12)0.125 


For  Mission  Capability; 
xi  >=  66  andx;  <=100 
X2  >=26  and  X2  <=65 
X3  >=  66  and  xj  <=100 
X4  >=  26  and  X4  <=65 
xj  >=  61  and  x,  <=100 


For  Proposal  Risk; 
x<s>=61  andx,}<=100 
X7  >=  0  and  X7  <=60 
X8>=  61  and x,v  <=  1 00 

x?>=61  andx?  <=100 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 
(Subfactor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Sub factor  TMD  is  within  green  color  rating  range) 


(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

x/o  >=  0  and  xio  <=60  (CR  is  within  moderate  adjectival  rating  range) 


For  Past  Performance; 

xu  >=  3 1  and  xu  <=60  (PP  is  within  satisfactory  adjectival  rating  range) 
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With  some  little  changes,  this  formula  is  adapted  to  Airbus  to  be  able  to  do  the 
same  analysis.  The  model  will  be  solved  twice,  once  for  maximization  and  once  for 
minimization,  to  reach  the  results  for  possible  best/worst  scenario  overall  scores  for 
Boeing  and  Airbus. 

To  solve  the  model,  Microsoft  Excel  2003 ’s  Solver  tool  is  used  and  a  snapshot  of 
the  spreadsheet  is  also  presented  in  the  Appendix  section  of  this  study. 

The  model  in  the  second  sensitivity  analysis  is  adapted  to  Airbus  by  changing  the 
IFARA  rating  from  79  to  90  and  the  life  cycle  cost  (LCC)  rating  from  39.12  to  39.80  in 
the  objective  function  for  overall  score  (since  these  results  are  evaluated  directly  as 
numerical,  they  are  placed  as  constant  values  in  the  model).  Also,  changing  the  ranges  for 
xio  (cost  risk  sub  factor)  is  necessary,  because  Airbus  received  a  'low'  adjectival  rating 
instead  of  a  'moderate'  rating  (Boeing’s  CR  rating)  for  this  subfactor.  The  new  range  for 
xio  is  61  to  100  instead  of  0  to  60.  The  model  formulation  for  Airbus  is  also  presented  in 
the  Appendix  section  of  this  study. 

While  solving  for  maximization,  all  the  ratings  (decision  variables)  got  the  highest 
possible  points  in  their  range  and  while  solving  for  minimization,  all  the  ratings  got  the 
lowest  possible  points  in  their  range. 

The  overall  scores  for  Boeing  are  74.309  as  the  maximum  and  43.609  as  the 
minimum.  For  Airbus,  the  scores  are  77.735  as  the  maximum  and  48.085  as  the 
minimum.  The  ranges  for  max/min  overall  scores  are  30.70  for  Boeing  and  29.65  for 
Airbus.  The  max/min  scores  for  Airbus  are  both  almost  4  point  higher  than  Boeing’s 
scores,  but  their  scores  are  overlapping  for  26.22  points.  These  results  are  also  presented 
in  Table  32. 


VERSION  2 

BOEING 

AIRBUS 

DIFFERENCE  OF  BOEING/AIRBUS 

MAX 

74.309 

77.735 

3.426 

MIN 

43.609 

48.085 

4.476  | 

RANGE 

30.7 

29.65 

Table  32.  Summary  Results  Table  for  Sensitivity  Analysis  Version  Two. 
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The  sensitivity  of  overall  scores  for  both  alternatives  is  very  similar.  This  is 
because  while  using  the  color  and  adjectival  rating  methods  for  evaluation,  the  scores  for 
the  two  alternatives  are  very  similar.  There  are  just  three  differences  -  one  in  the  IFARA, 
one  in  the  LCC,  and  one  in  the  cost  risk  rating.  Only  three  out  of  the  thirteen  criteria  that 
are  affecting  the  overall  score  are  different,  and  these  are  only  slightly  different.  In 
addition,  since  these  three  factors  are  also  multiplied  by  small  weights,  they  make  an 
even  smaller  difference  in  the  overall  score.  Another  important  finding  from  these  results 
is  the  high  percentage  (almost  85%;  26.22/30.7)  of  overlap  between  the  ranges  of 
possible  points  for  both  alternatives.  This  overlapping  area  means  that  it  is  possible  for 
both  alternatives  to  win  the  contract  whenever  the  overall  scores  for  both  are  in  this 
overlapping  range.  Logically,  fewer  ranges  (rating  groups)  should  cause  a  larger  overlap 
region,  supporting  the  notion  that  the  less  sensitive  the  method,  the  more  likely  it  could 
undervalue  small  differences.  Since  the  model’s  second  version  had  a  smaller  number  of 
color  and  adjectival  ratings,  the  evaluation  results  become  even  less  sensitive  to  the  small 
differences. 

D.  THIRD  SENSITIVITY  ANALYSIS 

For  the  third  part  of  the  sensitivity  analysis,  the  weights  that  the  authors  have  used 
to  reflect  the  relative  importance  of  factors  and  subfactors  will  be  changed,  while  keeping 
original  model  ratings  (version  1  of  the  model)  as  constant,  to  analyze  the  effects  of 
weights  on  results.  In  this  analysis  part,  all  the  weights  (for  factors  and  subfactors)  will  be 
changed  to  analyze  the  effects  on  overall  scores.  The  authors’  model  for  the  third  part  of 
the  sensitivity  analysis  will  be  solved  for  maximization  and  minimization  to  understand 
the  effective  change  range  for  weights.  The  effects  of  weight  change  will  be  analyzed  for 
each  alternative  four  times,  which  are  the  following:  weights  for  maximum  overall  score 
in  the  color  rating  method,  weights  for  minimum  overall  score  in  the  color  rating  method, 
weights  for  maximum  overall  score  in  the  WSM  rating  method,  and  weights  for 
minimum  overall  score  in  the  WSM  rating  method.  These  four  analyses  will  be  done  for 
each  alternative  separately  and  the  fonnulation  will  have  small  changes  for  each  iteration. 
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In  the  model,  all  the  factor  and  subfactor  weights  will  be  the  decision  variables. 
The  objective  function  will  be  an  overall  score  function  that  consists  of  all  subfactor 
ratings  and  their  weights,  and  the  constraints  will  be  the  relative  importance  of  subfactors 
and  factors  with  equality  of  factor/sub  factor  weight  totals  to  one  for  each  factor/subfactor 
level.  Since  all  the  decision  variables  will  be  in  a  positive  number  range,  there  will  not  be 
a  non-negativity  constraint  (redundant  constraint-this  constraint  does  not  add  any  value  to 
the  model).  Also,  the  model  will  be  solved  as  a  non-linear  model. 

MODEL(S)  FOR  BEST/WORST  RATINGS  FOR  WEIGHT  CHANGES 
(FOR  BOEING  FOR  COLOR  RATING  METHOD  IN  VERSION  3) 

Decision  Variables: 


Xi  :  Mission  Capability  (MC)  sub  factor  Key  System  Requirements  (KSR)  weight. 

x2  :  MC  subfactor  System  Integration  and  Software  (SIS)  weight. 

x3  :  MC  subfactor  Product  Support  (PS)  weight. 

x4  :  MC  subfactor  Program  Management  (PM)  weight. 

x5  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  weight. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  weight. 

x7  :  PR  subfactor  System  Integration  and  Software  (SIS)  weight. 

x8  :  PR  subfactor  Product  Support  (PS)  weight. 

x<>  :  PR  subfactor  Program  Management  (PM)  weight. 

x/o  :  Cost/Price  subfactor  Life  Cycle  Cost  (LCC)  weight. 

Xu  :  Cost/Price  subfactor  Cost  Risk  (CR)  weight. 

x  12  :  Mission  Capability,  Proposal  Risk  and  Past  Performance  factors’  weight. 

xI3  :  Cost/Price  and  IF ARA  factors’ weight. 

Objective  Function  (Non-linear  Model): 


Maximize  Overall  Score  =  [(87.5  X/  +  62.5  x2+  87.5  x3  +  62.5  x4+  83.3  X5)]  x32  + 
or  [(83.3  Xf,  +  50  x2  +  83.3  x8  +  83.3  x^)]  x32  + 

Minimize  58.35  x12  +  83.2  x13+  (50  xn  +  39.12  xI0)  xI3 


Constraints: 

For  Mission  Capability; 

xi  >  x2  (KSR  is  more  important  than  SIS) 

x'2  >  x3  (SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(PM  is  more  important  than  TMD) 
(all  of  the  weights  are  bigger  than  0) 
(total  of  weights  are  equal  to  1) 


x3>x4 
x4  >  X5 
x5>  0 
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For  Proposal  Risk; 

X6>  Xj 
X7>Xs 
X8>Xg 

x9>0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(all  of  the  weights  are  bigger  than  0) 
X(,  +  Xy  +  x8  +  x9=  1  (total  of  weights  are  equal  to  1) 


For  Cost/Price; 

Xio  >  Xu 

Xu  >  0 


(LCC  is  more  important  than  CR) 
(all  of  the  weights  are  bigger  than  0) 


Xio  +  Xu  =  1  (total  of  weights  are  equal  to  1) 


For  Factor  Level; 


Xl2  >  x,3 

Xl3>  0 


(MC,PR  and  PP  are  more  important  than  CR  and  IFARA) 
(all  of  the  weights  are  bigger  than  0) 
(total  of  weights  are  equal  to  1) 


The  formulation  above  is  the  base  formulation  for  the  third  part  of  the  sensitivity 
analysis.  It  is  the  fonnulation  for  Boeing’s  color  rating  in  version  one.  In  this  analysis 
part,  the  model  is  no  longer  linear  because  of  the  multiplication  of  some  decision 
variables  which  causes  the  authors’  model  to  become  a  non-linear  model  (MC,  PR  and 
CR  subfactor  weights  are  multiplied  by  factor  level  weights  and  then  the  model  becomes 
non-linear). 

In  all  of  the  iterations  of  the  model,  the  decision  variables  and  the  constraints  will 
not  be  changed.  The  only  differences  will  be  made  in  the  objective  function  and  it  will  be 
the  change  of  some  constant  values  for  the  color  rating  part  for  Airbus  (the  differences 
between  the  models  for  Boeing  and  Airbus  are  highlighted  below),  and  the  change  of  all 
the  constant  values  for  the  WSM  rating  method  (for  Boeing  and  Airbus).  All  three 
objective  functions  are  presented  below  (not  the  whole  mathematical  formulation)  and 
also  full  and  separate  versions  of  the  formulations  for  each  iteration  can  be  found  in  the 
Appendix  section  of  this  study. 

To  solve  the  model,  Microsoft  Excel  2003 ’s  Solver  tool  is  used  and  a  snapshot  of 
the  spreadsheet  is  also  presented  in  the  Appendix  section  of  this  study. 


106 


Objective  Function  for  Airbus  Color  Rating  Weights  Change  Effects: 


Maximize  Overall  Score  =  [(87.5  xi  +  62.5  x?+  87.5  X3+  62.5  X4+  83.3  X5)]  xn  + 

or  [(83.3  X6  +  50  X7  +  83.3  x«  +  83.3  x?)]  xn  + 

Minimize  58.35  X12  +  92  xu+  (83.3  x//  +  39.8  x/o)  xn 

Objective  Function  for  Boeing  Numerical  Rating  Weights  Change  Effects: 

Maximize  Overall  Score  =  [(93  x/  +  64  xi+  92  x.?  +  63  x^  +  94  xj)]  x/2  + 

or  [(94  xg  +  57  X7  +  97  xs  +  94  X9)]  X;2  + 

Minimize  59  xn  +  83.2  x;i+  (58  x//  +  39.12  xjo)  xjs 

Objective  Function  for  Airbus  Numerical  Rating  Weights  Change  Effects: 

Maximize  Overall  Score  =  [(8 1  x/  +  60  x?  +  90  x(+  58  x4+  85  X5)]  xn  + 

or  [(85  X6  +  57  X7  +  91  xg  +  87  xgj\  xn  + 

Minimize  56  x/2  +  92  X/5+  (70  Xu  +  39.8  xw)  x/_? 


After  solving  all  these  iterations  to  analyze  the  overall  score  sensitivity  to  weight 
change,  the  overall  scores  for  Boeing  are  75.30  as  the  maximum  and  65.04  as  the 
minimum  for  the  color  rating  method  and  80.75  as  the  maximum  and  67.38  as  the 
minimum  for  the  WSM  method.  The  overall  scores  for  Airbus  are  75.97  as  the  maximum 
and  66.95  as  the  minimum  for  the  color  rating  method  and  73.44  as  the  maximum  and 
66.18  as  the  minimum  for  the  WSM  method.  The  results  are  presented  in  Table  33; 


COLOR  MAX 

COLOR  MIN 

WSM  MAX  |  WSM  MIN 

BOEING 

75.30 

65.04 

80.75  I  67.38 

ORIGINAL  68.02  |  ORIGINAL  72.15 

AIRBUS 

75.97  1 

I  66.95 

73.44  |  66.18 

ORIGINAL  70.84 

ORIGINAL  70.14 

Table  33.  Summary  Results  Table  for  Sensitivity  Analysis  Version  Three. 


Also,  the  change  range  for  overall  scores,  which  is  due  to  weight  changes,  is  very 
similar  for  Boeing  (10.26)  and  for  Airbus  (9.03)  in  the  color  rating  method.  However,  in 
the  WSM  method  the  change  range  difference  is  significant,  as  it  is  13.38  for  Boeing  and 
7.26  for  Airbus.  Because  Boeing’s  ratings  for  most  of  the  subfactors  are  higher  than 
Airbus  in  WSM  method,  Boeing  is  more  sensitive  to  weight  changes  in  WSM  method. 
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These  results  also  show  the  difference  between  the  color  rating  method  and  the  WSM 
method,  as  the  color  rating  method  is  less  competent  than  the  WSM  method  to  reflect  the 
little  changes,  the  differences  in  ratings  are  hide  in  color  ratings  and  both  Boeing  and 
Airbus  change  ranges  become  similar. 

In  this  part  of  the  sensitivity  analysis,  the  authors  also  wanted  to  analyze  the 
overall  scores  for  a  special  case.  They  implemented  Boeing’s  best  case  scenario  weights 
to  calculate  the  overall  score  for  Airbus  in  the  same  scenario.  The  Microsoft  Excel 
snapshot  of  this  scenario  is  shown  in  the  Appendix  section  of  this  study.  For  the  color 
rating  method,  the  results  were  75.30  for  Boeing  and  75.56  for  Airbus,  which  means  with 
some  cost,  cost  risk  rating  and  IFARA  advantage  Airbus  won  with  the  color  rating  model 
even  with  Boeing’s  best  scenario  weights.  For  the  numerical  rating,  the  results  were 
80.75  for  Boeing  and  73.44  for  Airbus,  which  results  in  a  significant  advantage  for 
Boeing.  Table  34  below  summarizes  these  results. 


BOEING'S  BEST  RESULT  RATINGS  VS.  AIRBUS  WITH  SAME  WEIGHTS 


COLOR  |  NUMERICAL 

BOEING 

75.30 

AIRBUS 

Table  34.  Summary  Results  Table  for  Special  Case:  Boeing’s  Best  Case  Scenario. 

The  authors  also  implemented  the  best  case  scenario  weights  for  Airbus  to 
calculate  Boeing’s  overall  score  in  that  scenario.  The  Microsoft  Excel  snapshot  of  this 
scenario  is  also  shown  in  the  Appendix  section  of  this  study.  For  the  color  rating  method, 
the  results  were  71.00  for  Boeing  and  75.97  for  Airbus,  which  results  in  a  significant 
advantage  for  Airbus.  For  the  numerical  rating,  the  results  were  the  same  as  the  previous 
scenario  and  both  alternatives’  best  results  because  the  best  result  weights  were  the  same 
for  both  alternatives  in  the  numerical  method.  A  summary  of  these  results  is  shown  in 
Table  35. 
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AIRBUS'S  BEST  RESULT  RATINGS  VS.  BOEING  WITH  SAME  WEIGHTS 

COLOR 

NUMERICAL  j 

BOEING 

71.00 

AIRBUS 

73.44  | 

Table  35.  Summary  Results  Table  for  Special  Case:  Airbus’s  Best  Case  Scenario. 

E.  CONCLUSIONS 

The  sensitivity  analysis  to  understand  the  effects  of  rating  change  within  color 
ranges  on  both  versions  of  the  authors’  study  produced  identical  results.  This  is  due  to  the 
fact  that  while  using  color  and  adjectival  rating  methods  for  evaluation,  the  scores  for  two 
alternatives  are  very  similar.  There  are  just  three  differences  -  one  in  the  IFARA,  one  in 
the  LCC,  and  one  in  the  cost  risk  rating.  Only  three  out  of  the  thirteen  criteria  that  are 
affecting  the  overall  score  are  slightly  different.  In  addition,  since  these  three  factors  are 
also  multiplied  by  small  weights,  they  make  an  even  smaller  difference  in  the  overall 
score.  Another  important  finding  of  the  results  of  rating  change  within  color  ranges 
effects  analysis  for  the  two  models  is  the  high  percentage  of  overlap  between  the  ranges 
of  possible  points  for  both  alternatives.  This  overlapping  area  means  that  it  is  possible  for 
either  alternative  to  win  the  contract  whenever  the  overall  scores  for  both  are  in  this 
overlapping  range.  Also,  the  overlapping  area  allows  subjectivity  in  the  evaluation  of 
offeror  proposals.  The  difference  between  these  sensitivity  analysis  implementations  on 
the  two  versions  of  the  authors’  study  is  the  results  of  range  changes  (elimination  of  some 
ratings  lead  to  redetermination  of  ranges).  Since  the  ranges  get  bigger  in  the  second 
version,  the  effects  of  rating  change  get  bigger  as  expected  and  the  amount  of  subjectivity 
(with  growing  overlapping  ranges)  allowed  gets  increased. 

The  sensitivity  analysis  to  understand  the  effects  of  weight  (factor  and  subfactor 
weights  all  together)  changes  on  the  overall  scores  produced  interesting  results.  The 
change  ranges  for  overall  scores  were  very  similar  for  Boeing  and  for  Airbus  in  the  color 
rating  method.  However,  in  the  WSM  method  the  change  range  difference  was 
significant.  These  results  show  how  the  color  rating  method  is  less  sensitive  than  the 
numerical  method  (less  competent  to  reflect  the  little  changes). 
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The  authors  finished  their  sensitivity  analysis  with  a  special  case  scenario 
analysis.  Boeing’s  best  case  scenario  weights  implemented  to  Airbus  and  results  for  the 
color  rating  method  lead  to  an  Airbus  award  even  in  Boeing’s  best  case  scenario.  In  that 
case,  it  shows  that  even  with  higher  numerical  scores,  Boeing  will  not  be  able  to  present 
its  better  value  (better  value  than  Airbus)  under  the  color  rating  method.  In  WSM 
method,  Boeing’s  best  case  scenario  weights  implemented  to  Airbus  lead  to  a  significant 
overall  score  advantage  for  Boeing.  Airbus’s  best  case  scenario  weights  implemented  to 
Boeing  resulted  with  significant  advantage  for  Airbus  in  color  rating  method,  however 
the  results  were  significantly  advantageous  for  Boeing  in  WSM  method  (even  in  Airbus’s 
best  case  scenario). 
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VIII.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SUMMARY 

The  source  selection  phase  of  government  acquisitions  involves  many  interrelated 
tasks.  Even  though  objective  factors  and  subfactors  are  used  in  the  source  selection 
phase,  subjectivity  is  inevitable  in  government  contracting.  This  is  because  the  Federal 
Acquisition  Regulation  (FAR)  does  not  provide  any  clear  guidance  to  evaluate  proposals 
and,  thus,  the  source  selection  decision  is  usually  made  based  on  some  evaluations 
“conducted  using  any  rating  method  or  a  combination  of  methods,  including  color  or 
adjectival  ratings,  numerical  weights,  and  ordinal  rankings.”143  Besides,  proposals  are 
evaluated  by  different  personnel  and  conclusions  may  differ,  even  for  the  same  factor, 
significantly.144  Therefore,  considering  the  high  complexity  in  the  nature  of  government 
acquisitions,  the  comparison  of  proposals  and  source  selection  decision  involve 
judgmental  evaluations.145 

Considering  the  fact  that  available  resources  to  the  government  are  becoming 
scarcer,  subjectivity  should  be  diminished  in  contracting  and  agencies  should  find  ways 
to  get  the  best  value  from  any  procurement.  Quantitative  methods  and  information 
technology  tools  can  be  utilized  to  augment  objectivity  in  source  selection  decision 
making. 

The  aim  of  this  project  is  to  show  the  effects  of  the  choice  of  model  (method) 
used,  as  well  as  decisions  made  during  the  modeling  and  assessment  process.  These 
choices  may  contrast  with  the  fairness  and  the  best  value  part  of  government  procurement 
processes.  Fair  to  whom  and  best  value  for  whom  were  the  very  first  questions  in  the 


143  FAR  15.305. 

144  Cibinic  and  Nash,  Formation  of  Government  Contracts,  821. 

145  Templin  and  Noffsinger,  “An  Assessment,”  38. 
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authors’  minds  when  they  started  this  project.  The  authors  tried  to  show  the  effects  of 
subjectivity  involved  in  the  process  of  selection  by  using  a  simplified  current  real-life 
program. 

The  authors  chose  the  USAF’s  KC-X  program  as  a  case  study  for  their  research 
questions.  There  are  some  specific  reasons  for  their  choice  of  that  program.  The  authors 
focused  on  an  Air  Force  procurement  program  in  which  color  ratings  were  used,  because 
in  their  perspective,  the  Air  Force’s  color  rating  is  one  of  the  most  subjective  techniques 
used  in  source  selection.  The  authors  decided  to  analyze  a  program  that  was  popular  and 
current.  The  authors  thought  that  they  could  have  a  variety  of  comments  from  different 
viewpoints.  With  a  current  program,  they  hoped  to  have  more  insight  information,  which 
was  publicly  available,  about  the  program.  After  their  initial  research  on  the  program,  the 
authors  realized  that  the  two  alternatives  offered  very  different  specifications  and 
capabilities  for  the  same  program.  This  showed  the  authors  the  difference  of  the  two 
offerors’  point  of  view.  The  authors  thought  that  this  program  might  be  a  good  sample  for 
their  study  because  of  the  potential  effects  of  the  choices  made  for  and  during  the 
evaluation.  Another  factor  that  made  the  choice  even  more  interesting  for  this  study  was 
the  high  potential  for  the  award  to  be  protested.  Experts  of  the  defense  industry  were 
expecting  a  protest  from  the  loser.  Since  the  proposed  KC-X  alternatives  have  very 
different  strengths  in  capabilities,  results  will  not  easily  satisfy  the  offeror  who  loses  the 
program  award. 

The  source  selection  phase  in  government  contracting  involves  multi-criteria 
decision  making  which  has  to  deal  with  many  critical  factors  such  as  cost,  technical 
capabilities,  and  the  past  performance  of  contractors.  The  contracting  processes  used  in 
government  acquisitions  have  been  built  to  respond  to  various  government  procurement 
needs  by  evaluating  these  factors. 

In  Chapter  II  an  overview  of  acquisitions,  general  summary  information  about 
acquisition  processes  was  provided.  First,  contracting  methods  were  discussed  and  then, 
the  evaluation  and  selection  processes  were  defined  briefly.  Since 
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this  project  focused  on  the  Air  Force’s  color  rating  method,  the  Air  Force  source 
selection  was  examined  a  little  deeper  on  categories  of  Air  Force  acquisitions,  steps  in 
Air  Force  source  selection,  evaluation  activities,  and  award  activities. 

Chapter  III  provided  an  overview  of  decision-making  support  techniques,  the 
authors  gave  some  background  infonnation  about  the  available  techniques  to  solve  a 
multi-criteria  problem  since  the  study  case  (the  Air  Force’s  KC-X  program)  is  a  major 
weapon  systems  procurement  and  has  more  than  one  criterion  to  determine  while  trying 
to  come  up  with  the  best  value  product  for  the  government.  Multi-criteria  decision 
analysis  (MCDA)  offers  numerous  methodologies  and  schools  of  thought  for  multi¬ 
criteria  modeling.  The  approaches  vary  from  very  simple  ones  to  extremely  complex 
optimization  models. 

The  authors  used  the  weighted  sum  method  (WSM)  in  their  study.  They  preferred 
a  technique  that  was  simple  to  understand  by  anyone  who  has  no  multi-criteria  decision 
making  (MCDM)  background.  WSM  is  one  of  the  very  basic  models/techniques  used  in 
situations  where  more  than  one  criterion  is  involved  and  the  relative  importance  of  each 
criterion  is  different  from  each  other.  The  model’s  biggest  strength  is  its  simplicity  in 
application. 

Chapter  IV  discussed  the  USAF’s  KC-X  Next  Generation  Tanker  Planes 
Program,  the  authors  gave  some  background  information  about  the  Air  Force’s  KC-X 
program.  They  used  this  program  in  their  model  to  show  the  subjectivity  involved  in  the 
source  selection  proposal  evaluation  part  of  the  Air  Force’s  acquisitions.  Introductory 
information  about  the  program,  the  requirements  of  the  program,  the  competition 
between  the  two  offerors,  and  lastly  some  specific  information  (to  roughly  compare  the 
two  alternatives)  about  each  proposed  aircraft  were  presented  in  this  chapter. 

In  the  KC-X  Tanker  Replacement  Program,  which  is  recently  the  most  significant 
procurement  program  in  the  Air  Force,  the  Air  Force  used  its  current  source  selection 
method,  namely  the  color  rating  method,  and  awarded  the  contract  to  the  Airbus/Northrop 
Grumman  team.  After  debriefings  were  presented,  the  fact  emerged  that  the  ratings  of 
both  offerors  were  almost  the  same.  Consequently,  long  debates  have  begun  between 
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Boeing  and  the  Air  Force  with  Boeing  filing  a  protest  with  the  Government 
Accountability  Office  (GAO)  for  review  of  the  award  decision.  The  similar  color  ratings 
led  to  the  protest  exercised  in  the  KC-X  program  and  in  some  previous  contract  awards; 
therefore  in  the  authors’  model  this  was  one  of  the  focus  areas. 

In  Chapter  V  about  model  construction,  two  models  that  were  built  within 
Microsoft  Excel  were  presented  in  two  versions  to  show  the  different  characteristics  of 
two  types  of  methods  that  can  be  used  in  the  source  selection  phase  in  government 
acquisitions.  These  methods  are  color  rating,  which  is  the  current  method  used  by  the 
U.S.  Air  Force,  and  the  quantitative  weighted  sum  method.  The  models  were  established 
using  some  data  from  the  Air  Force’s  current  KC-X  Tanker  Replacement  Program  and 
proposals  of  its  two  offerors,  Boeing  and  Northrop/Airbus.  In  Chapter  VI  that  discusses 
the  analysis  and  results,  simplified  KC-X  program  data  was  implemented  in  these 
models.  Two  offerors’  color  rating  data  that  are  provided  from  publicly  available  sources 
were  used  in  the  models. 

In  Chapter  VII  that  presents  the  sensitivity  analysis,  the  authors  investigated  how 
modeling  choices  might  affect  the  results  that  were  found  in  the  previous  chapter.  A 
sensitivity  analysis  was  done  to  understand  the  effects  and  limits  of  subjectivity  in  the 
authors’  models.  Three  different  forms  of  sensitivity  analysis  were  done.  In  the  first  two, 
the  numerical  ratings  for  the  color  and  adjectival  ratings  were  changed  within  their  ranges 
to  determine  the  best/worst  possible  overall  scores  for  each  alternative  in  the  first  and 
second  versions  of  the  authors’  model.  In  the  third  one,  the  weights  were  changed,  while 
keeping  original  model  ratings  as  constant,  to  analyze  the  effects  of  weights  on  the 
results. 

B.  CONCLUSIONS 

In  the  authors'  study,  with  their  arbitrary  numerical  ratings  chosen  to  be  within  the 
ranges  of  the  color  ratings  given  in  the  KC-X  program,  they  were  able  to  show  that  when 
they  used  the  USAF’s  color  rating  method,  Airbus  could  have  won  the  contract,  but  when 
they  used  the  WSM  method,  Boeing  could  have  won  the  contract.  The  implication  under 
these  results  is  that  when  there  are  small  but  important  differences  between  offerors,  the 

114 


color  rating  method  is  potentially  incompetent  to  reflect  those  differences.  However,  the 
quantitative  weighted  sum  method  is  capable  of  handling  those  little  details  and  providing 
more  accurate  results. 

As  explained  previously,  in  the  authors'  models,  version  two  differed  from  version 
one  based  on  some  changes  to  the  color/adjectival  ratings  used  and  their  numerical 
ranges.  Initially,  the  color/adjectival  ratings  that  reflected  the  lowest  values  for 
factors/sub  factors  were  eliminated  and  then,  the  numerical  ranges  to  which  the  color  and 
adjectival  ratings  were  related  to  were  extended  or  narrowed.  The  arbitrarily  chosen 
numerical  ratings  were  then  adjusted  so  that  they  still  fell  within  the  ranges  of  the  color 
ratings  given  within  the  KC-X  program.  The  second  version  of  the  authors'  model,  which 
suggested  a  different  way  to  evaluate  both  offerors,  produced  the  same  significant  result 
as  was  obtained  in  the  first  version. 

This  part  of  the  study  supported  the  authors'  argument  that  the  quantitative 
weighted  sum  method  is  more  competent  than  the  color  rating  method  in  reflecting  the 
little  but  important  differences  between  offerors. 

In  Chapter  VII  concerning  sensitivity  analysis,  first  the  authors  investigated  how 
modeling  choices  might  affect  previous  results  that  they  found.  They  performed  a 
sensitivity  analysis  to  answer  the  research  question:  “How  sensitive  are  the  overall  WSM 
scores  to  different  numerical  ratings  within  each  color  range?”  While  keeping  the  color 
rating  ranges  constant,  the  authors  varied  the  numerical  ratings  within  the  given  ranges 
for  the  color  rating  that  was  assigned  to  see  how  much  better  or  worse  the  overall  score 
could  be  while  still  corresponding  to  the  same  color  rating. 

The  sensitivity  of  overall  scores  for  both  alternatives  was  very  similar.  This  is 
because  while  using  color  and  adjectival  rating  method  for  evaluation,  the  scores  for  two 
alternatives  were  very  similar.  There  were  just  three  differences  -  one  in  the  IFARA,  one 
in  the  LCC,  and  one  in  cost  risk  rating.  Only  three  out  of  the  thirteen  criteria  that  were 
affecting  the  overall  score  were  different,  albeit  only  slightly  different.  In  addition,  since 
these  three  factors  were  also  multiplied  by  small  weights,  they  made  an  even  smaller 
difference  in  the  overall  score.  Another  important  finding  of  these  results  is  the  high 
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percentage  of  overlap  between  the  ranges  of  possible  final  scores  for  both  alternatives. 
This  overlapping  area  means  that  it  is  possible  for  either  alternative  to  win  the  contract 
whenever  the  overall  scores  for  both  are  in  this  overlapping  range. 

Secondly,  the  sensitivity  analysis  used  in  the  first  part  was  used  in  the  second 
model  as  well.  The  authors  examined  the  results  of  the  analysis  to  understand  the  effects 
of  the  choice  of  color  rating  ranges  on  the  overall  scores.  The  research  question  here  was 
“How  sensitive  is  the  overall  numerical  score  to  different  numerical  ratings  within  each 
color  range  with  a  diminished  number  of  color  and  adjectival  ratings  with  appropriately 
adjusted  color  rating  ranges?” 

Similarly  with  the  first  sensitivity  analysis,  the  overall  score  sensitivities  for  both 
alternatives  were  very  similar.  It  is  because  while  using  the  color  and  adjectival  rating 
method  for  evaluation,  the  scores  for  the  two  alternatives  were  very  similar  and  also  with 
the  multiplier  effect  of  weights  these  differences  were  getting  less  significant.  Another 
important  finding  of  these  results  is  the  high  percentage  of  overlap  between  the  ranges  of 
possible  final  scores  for  both  alternatives.  This  overlapping  area  means  that  it  is  possible 
for  either  alternative  to  win  the  contract  whenever  the  overall  scores  for  both  are  in  this 
overlapping  range.  Logically,  fewer  ranges  (rating  groups)  should  cause  a  larger  overlap 
region,  supporting  the  notion  that  the  less  sensitive  the  method,  the  more  likely  it  could 
undervalue  small  differences.  Since  the  authors’  second  version  of  their  model  had  less 
number  of  color  and  adjectival  ratings,  the  evaluation  results  became  even  less  sensitive 
to  the  small  differences. 

The  sensitivity  analysis  to  understand  the  effects  of  rating  change  within  the  color 
range  on  both  versions  of  the  authors’  study  produced  very  similar  results,  namely  a  high 
percentage  of  overlap  for  final  score  ranges  for  both  alternatives  in  both  versions.  The 
overlapping  area  means  that  it  is  possible  for  both  alternatives  to  win  the  contract 
whenever  the  overall  scores  for  both  are  in  this  overlapping  range.  The  overlapping 
areabrings  about  the  subjectivity  issue  to  the  evaluation  of  offeror  proposals,  because  as 
the  ranges  got  bigger  in  the  second  version,  the  effects  of  the  rating  change  got  bigger  as 
expected  and  the  subjectivity  (with  a  growing  overlapping  range)  involved  increased  as 
well. 
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In  the  third  part  of  the  authors’  sensitivity  analysis,  the  weights  were  changed  in 
both  the  color  rating  method  and  the  WSM  method,  while  the  original  model  ratings  were 
kept  constant,  to  analyze  the  effects  of  weights  on  the  results.  In  this  analysis  part,  all  the 
weights  (for  factors  and  subfactors)  were  varied  to  analyze  the  effects  on  overall  scores. 
The  sensitivity  analysis  to  understand  the  effects  of  weight  changes  (including  all  factor 
and  subfactor  weights)  on  the  overall  scores  produced  interesting  results.  The  range  of 
changes  for  overall  scores  was  very  similar  for  Boeing  and  for  Airbus  in  the  color  rating 
method.  However,  in  the  WSM  method  the  change  range  difference  was  significant. 
These  results  were  more  substantiating  data  for  the  authors’  study  to  show  how  the  color 
rating  method  is  less  sensitive  than  the  WSM  method  (less  competent  to  reflect  the  small, 
but  potentially  important  differences  between  alternatives). 

In  the  last  part  of  the  sensitivity  analysis,  the  authors  also  wanted  to  analyze  the 
overall  scores  for  two  special  cases.  First,  they  solved  for  the  weights  that  would  give 
Boeing  the  best  possible  overall  score.  Then,  they  implemented  Boeing’s  best  case 
scenario  weights  to  calculate  the  overall  score  for  Airbus  in  the  same  scenario.  For  the 
color  rating  method,  Airbus  won  even  with  Boeing’s  best  scenario  weights.  For  the  WSM 
rating,  the  results  were  significantly  better  for  Boeing.  The  authors  also  solved  for 
Airbus’s  best  case  scenario  weights  too  and  used  them  to  calculate  Boeing’s  overall  score 
in  the  same  scenario.  The  results  were  very  similar  to  the  first  special  case,  as  Airbus  won 
in  with  the  color  rating  method  even  with  a  significant  advantage.  However,  for  the  WSM 
method  Boeing  was  far  better  than  Airbus. 

It  is  necessary  to  remember  that  in  this  study,  the  authors  had  only  the  color 
ratings  that  were  available,  and  therefore,  they  converted  these  color  ratings  to  numbers 
and  arbitrarily  chose  the  ratings  used  in  the  WSM  model  to  fall  within  the  same  color 
rating  ranges.  All  the  results  came  from  the  models  where  the  authors  used  those  arbitrary 
numbers.  The  purpose  of  the  study  was  not  to  say  who  should  have  won  the  KC-X 
contract.  The  actual  evaluation  of  proposals  was  done  by  a  team  consisting  of  many 
experts  and  the  evaluation  period  was  almost  a  year  even  for  such  a  big  team  (the  authors 
did  not  have  enough  information,  time,  and  intent  to  do  the  same  evaluation  in  a  better 


117 


way).  The  authors  simply  wanted  to  show  that  the  contract  resulted  in  a  different  winner 
by  using  numbers  and  the  WSM  method  instead  of  the  color  and  adjectival  ratings 
method. 

The  color  rating  method  was  not  sensitive  enough  for  little  differences  in  the 
evaluation  ratings.  These  little  differences  in  each  factor  or  subfactor  ratings  may  be  in 
favor  of  one  alternative.  Thus,  by  using  the  color  rating  method  one  may  eliminate  all 
those  little  changes  and  consider  both  alternatives  with  the  same  color  ratings  and  then 
possibly  choose  the  wrong  one  (not  the  one  that  gives  the  best  value). 

The  authors’  sensitivity  analysis  also  showed  that  the  color  rating  range 
determination  is  very  critical  because  the  effects  of  these  ranges  on  the  results  are  pretty 
significant.  The  effects  of  color  rating  ranges  are  much  higher  than  the  effects  of  the 
weights  of  factors  and  subfactors.  Additionally,  the  first  (color  rating  range  analysis  on 
version  one)  and  second  sensitivity  (same  analysis  on  version  two)  showed  the  authors 
that  with  the  less  number  of  ratings  they  increased  the  ranges  and  as  a  result  they  made 
the  color  rating  method  even  less  sensitive  to  little  differences. 

The  third  part  of  the  sensitivity  analysis  results  showed  the  effects  of  weights  on 
overall  scores.  As  mentioned  before,  the  effects  of  weights  were  not  as  big  as  the  effects 
of  the  color  and  adjectival  rating  ranges.  One  of  the  biggest  reasons  for  the  weights  being 
less  effective  was  because  the  verbal  explanations  of  the  relationships  between  these 
weights  were  given  precisely  in  the  RFP,  so  there  was  not  much  flexibility  to  change  the 
weights. 

In  conclusion,  although  the  color  rating  method  is  currently  the  method  for  the  Air 
Force,  it  is  not  necessarily  competent  whenever  the  scores  of  the  proposed  options  are 
close.  In  some  cases  like  the  one  the  authors  analyzed,  the  color  rating  method  may 
eliminate  some  of  the  small  differences  (however,  when  aggregated  these  small 
differences  may  affect  the  overall  score  significantly)  and  may  lead  the  decision  maker  in 
the  wrong  direction.  The  government  has  different  responsibilities  than  the  private  sector, 
like  being  fair  all  the  time.  So,  when  seeking  the  best  value,  the  government  should  make 
a  fair  evaluation  of  the  proposals  and  explain  the  logic  of  their  decision.  With  the  color 
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rating  method,  the  authors  think  that  the  government  in  some  cases  may  not  be  able  to 
make  the  evaluation  fairly.  This  will  not  only  lead  to  some  trust  issues  but  also  lead  to 
protests  from  the  losers.  With  the  protests,  even  the  most  important  programs  can  be 
suspended  and  a  significant  amount  of  money  can  be  spent  in  resolving  the  issues. 

C.  RECOMMENDATIONS  AND  FUTURE  RESEARCH 

The  government’s  responsibility  is  not  only  managing  its  programs  successfully 
from  a  business  stand  point;  government  also  has  some  other  responsibilities  like  socio¬ 
economic  responsibilities.  The  government  must  act  fairly  in  all  contract  actions  and 
making  a  fair  evaluation  of  competing  proposals  is  one  of  these  responsibilities. 
Therefore,  the  government  must  be  consistent  in  its  evaluation  of  the  proposals  process 
and  precise  while  explaining  its  requirements  and  methods  for  the  evaluations.  The  color 
rating  method  is  not  a  very  competent  method  to  obtain  the  best  value,  but  it  is  clear  that 
it  gives  the  evaluator  some  space  for  flexibility  when  detennining  the  best  value  for  the 
government.  The  color  rating  method  may  end  up  eliminating  the  individual  evaluation 
differences  and  establish  a  consensus  on  the  score  for  individual  factors  and  subfactors, 
but  it  is  not  competent  enough  to  be  used  as  the  main  method  to  detennine  which  offeror 
will  be  awarded  the  contract.  Although  it  seems  that  the  color  rating  method  is  less 
subjective  since  it  eliminates  individual  evaluation  differences,  it  may  come  up  with 
subjective  results  when  used  as  the  main  method  for  evaluation  of  the  proposals  because 
of  the  elimination  of  small  differences  and  their  aggregate  effects  to  the  overall  score. 
When  the  color  rating  results  are  similar,  which  is  very  likely  to  happen,  it  is  hard  to 
justify  the  choice  that  has  been  made  as  a  government.  Whenever  the  offerors  are  not 
satisfied  with  the  results,  and  especially  when  they  feel  the  results  did  not  show  a  clear 
winner  (the  lack  of  clarity  in  the  evaluation),  they  are  very  likely  to  protest  the  award, 
which  leads  to  suspension  of  the  program  until  all  the  protests  are  resolved. 

Instead  of  using  a  purely  numerical  model  such  as  the  weighted  sum  method,  the 
color  rating  method  can  be  developed  again  with  an  increased  number  of  color  and 
adjectival  ratings  instead  of  four  or  five  ratings  in  order  to  increase  its  sensitivity  to  small 
differences.  A  more  extensive  numerical  study  could  be  performed  to  develop  a 
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recommendation  for  how  many  colors  to  use  (how  sensitive  one  wants  the  color  rating 
method  to  small  differences)  and  how  to  define  the  conditions  under  each  given  color 
(similar  to  what  is  in  place  now  for  the  red-yellow-green-blue  color  scheme). 

Another  study  could  be  done  to  understand  the  relationship  between  the  color 
rating  method  and  protests.  This  study  could  explore  whether  the  U.S.  Air  Force  was 
being  protested  more  or  less  before  they  started  using  the  color  rating  method  and  also 
determine  whether  protests  under  the  color  rating  method  are  more  common  when  the 
competing  proposals  were  given  similar  color  ratings. 
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APPENDIX 


Example  Snapshot  of  Version  One  Color  Rating  Spreadsheet 
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Example  Snapshot  of  Version  One  Weighted  Sum  Method  Spreadsheet 
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The  Snapshot  of  Version  One  Color  Rating  Method  Spreadsheet  for  Boeing 
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The  Snapshot  of  Version  One  Color  Rating  Method  Spreadsheet  for  Airbus 
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The  Snapshot  of  Version  One  Weighted  Sum  Method  Spreadsheet  for  Boeing 


A 

B 

C 

D 

E  I  F 

G 

H 

1 

J 

K 

L 

M 

N 

O 

1 

2 

MISSION  CAPABILITY 

WEIGHTS 

WEIGHTED 

RATINGS 

COST/PRICE 

SUBFACTORS 

WEIGHTS 

RATINGS 

WEIGHTED 

RATINGS 

IFARA 

RATING 

3 

SUBFACTORS 

1.79 

83.2 

4 

KSR 

93 

0  30 

27.90 

COST 

108  044 

0.60 

39.12 

23472 

83.2 

5 

SIS 

64 

0.25 

16.00 

RISK 

58 

0.40 

58.00 

23200 

6 

PS 

92 

020 

1840 

46  672 

7 

PM 

63 

0.15 

9.45 

8 

TMD 

94 

0.10 

940 

9 

81.15 

10 

11 

12 

PROPOSAL  RISK 

WEIGHTS 

WEIGHTED 

RATINGS 

PAST  PERFORMANCE 

RATING 

13 

SUBFACTORS 

SATISFACTORY 

59 

14 

KSR 

94 

0.35 

32.9 

59 

15 

SIS 

57 

0.30 

17.1 

16 

PS 

97 

0.20 

19.4 

17 

PM 

94 

0.15 

14.1 

18 

83.5 

19 

20 

21 

22 

FACTORS 

WEIGHTS 

RATING 

WEIGHTE 
D  RATING 

23 

24 

Mission  Capability 

0.250 

81.15 

20288 

25 

Proposal  Risk 

0250 

83.50 

20.875 

26 

Past  Performance 

0250 

59.00 

14.750 

27 

Cost/Price 

0.125 

46  67 

5.834 

28 

IFARA 

0.125 

83.20 

10  400 

29 

OVERALL  OFFEROR  RATING: 

72.147 

30 

125 


The  Snapshot  of  Version  One  Weighted  Sum  Method  Spreadsheet  for  Airbus 
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The  Comparison  Chart  Showing  Both  Offerors’  Ratings  Using  Both  Methods  in  Version  One 
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The  Snapshot  of  Version  Two  Color  Rating  Method  Spreadsheet  for  Boeing 
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The  Snapshot  of  Version  Two  Color  Rating  Method  Spreadsheet  for  Airbus 
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The  Snapshot  of  Version  Two  Weighted  Sum  Method  Spreadsheet  for  Boeing 
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The  Snapshot  of  Version  Two  Weighted  Sum  Method  Spreadsheet  for  Airbus 
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The  Comparison  Chart  Showing  Both  Offerors’  Ratings  Using  Both  Methods  in  Version  Two 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  within  Color  Ranges 

(For  Boeing  in  Version  1) 


Decision  Variables: 


xi  :  Mission  Capability  (MC)  subfactor  Key  System  Requirements  (KSR)  rating. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  rating. 

X3  :  MC  subfactor  Product  Support  (PS)  rating. 

X4  :  MC  subfactor  Program  Management  (PM)  rating. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  rating. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  rating. 

X7  :  PR  subfactor  System  Integration  and  Software  (SIS)  rating. 

xs  :  PR  subfactor  Product  Support  (PS)  rating. 

xg  :  PR  subfactor  Program  Management  (PM)  rating. 

xio  :  Cost/Price  subfactor  Cost  Risk  (CR)  rating. 

Xu  :  Past  Performance  (PP)  rating. 

Objective  Function  (Linear  Model): 


Maximize  Overall  Score  =  (0.3  xi  +  0.25  x?  +  0.2  x.?  +0.15  x4  +  0. 1  xs )  0.25  + 

or  (0.35  xg  +  0.3  X7  +  0.20  xs  +  0. 1 5  xg)  0.25  + 

Minimize  0.25  xu  +  0.125  *  83.2  + 

(0.40  xio  +  0.60  *  39.12)  0.125 

Constraints: 


For  Mission  Capability; 
xi  >=  76  andx;  <=100 
X2>=  51  and  X2  <=75 
X3  >=  76  and  xj  <=100 
x4  >=  5 1  and  x4  <=75 
xj  >=  61  andx5  <=100 

For  Proposal  Risk; 

X6  >=  67  and  xg  <=  1 00 
X7  >=  34  and  xz  <=66 
xg  >=  67  and  x,v  <=100 
xg  >=  67  and  xg  <=  1 00 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 

(Sub factor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Subfactor  TMD  is  within  green  color  rating  range) 

(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

xio  >=  34  and  xw  <=66  (CR  is  within  moderate  adjectival  rating  range) 

For  Past  Performance; 

xu>=  51  and  X//  <=66  (PP  is  within  satisfactory  adjectival  rating  range) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  within  Color  Ranges 

(For  Airbus  in  Version  1) 


Decision  Variables: 


xi  :  Mission  Capability  (MC)  subfactor  Key  System  Requirements  (KSR)  rating. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  rating. 

X3  :  MC  subfactor  Product  Support  (PS)  rating. 

X4  :  MC  subfactor  Program  Management  (PM)  rating. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  rating. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  rating. 

X7  :  PR  subfactor  System  Integration  and  Software  (SIS)  rating. 

xs  :  PR  subfactor  Product  Support  (PS)  rating. 

xg  :  PR  subfactor  Program  Management  (PM)  rating. 

xio  :  Cost/Price  subfactor  Cost  Risk  (CR)  rating. 

Xu  :  Past  Performance  (PP)  rating. 

Objective  Function  (Linear  Model): 


Maximize  Overall  Score  =  (0.3  xi  +  0.25  x?  +  0.2  x.?  +0.15  x4  +  0. 1  xs )  0.25  + 

or  (0.35  xg  +  0.3  X7  +  0.20  xs  +  0. 1 5  xg)  0.25  + 

Minimize  0.25  xu  +  0.125  *  92  + 

(0.40  x10  +  0.60  *  39.8)  0.125 

Constraints: 


For  Mission  Capability; 
xi  >=  76  andx;  <=100 
X2>=  51  and  X2  <=75 
X3  >=  76  and  xj  <=100 
x4  >=  5 1  and  x4  <=75 
xj  >=  61  andx5  <=100 

For  Proposal  Risk; 

X6  >=  67  and  xg  <=  1 00 
X7  >=  34  and  xz  <=66 
xg  >=  67  and  x,v  <=100 
xg  >=  67  and  xg  <=  1 00 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 

(Sub factor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Subfactor  TMD  is  within  green  color  rating  range) 

(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

xio  >=  67  and  xw  <=100  (CR  is  within  low  adjectival  rating  range) 

For  Past  Performance; 

xu>=  51  and  X//  <=66  (PP  is  within  satisfactory  adjectival  rating  range) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  within  Color  Ranges 

(For  Boeing  in  Version  2) 


Decision  Variables: 


xi  :  Mission  Capability  (MC)  subfactor  Key  System  Requirements  (KSR)  rating. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  rating. 

X3  :  MC  subfactor  Product  Support  (PS)  rating. 

X4  :  MC  subfactor  Program  Management  (PM)  rating. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  rating. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  rating. 

X7  :  PR  subfactor  System  Integration  and  Software  (SIS)  rating. 

xs  :  PR  subfactor  Product  Support  (PS)  rating. 

xg  :  PR  subfactor  Program  Management  (PM)  rating. 

xio  :  Cost/Price  subfactor  Cost  Risk  (CR)  rating. 

Xu  :  Past  Performance  (PP)  rating. 

Objective  Function  (Linear  Model): 


Maximize  Overall  Score  =  (0.3  xi  +  0.25  x?  +  0.2  x.?  +0.15  x4  +  0. 1  xs )  0.25  + 

or  (0.35  xg  +  0.3  X7  +  0.20  xs  +  0. 1 5  X9)  0.25  + 

Minimize  0.25  xj  1  +  0.125  *  79  + 

(0.40  xio  +  0.60  *  39.12)  0.125 

Constraints: 


For  Mission  Capability; 
x 1  >=  66  andx;  <=100 
X2  >=  26  and  X2  <=65 
X3  >=  66  andxs  <=100 
x4  >=  26  and  x4  <=65 
xj  >=  61  andx5  <=100 

For  Proposal  Risk; 
x<s>=61  andx,}<=100 
X7  >=  0  and  X7  <=60 
x,y  >=  6 1  and  x,y  <=  1 00 
xg>=6 1  andxp<=100 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 
(Sub factor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Subfactor  TMD  is  within  green  color  rating  range) 


(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

xio  >=  0  and  xw  <=60  (CR  is  within  moderate  adjectival  rating  range) 

For  Past  Performance; 

xu>=  31  and  Xu  <=60  (PP  is  within  satisfactory  adjectival  rating  range) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  within  Color  Ranges 

(For  Airbus  in  Version  2) 


Decision  Variables: 


xi  :  Mission  Capability  (MC)  subfactor  Key  System  Requirements  (KSR)  rating. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  rating. 

X3  :  MC  subfactor  Product  Support  (PS)  rating. 

X4  :  MC  subfactor  Program  Management  (PM)  rating. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  rating. 

X6  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  rating. 

X7  :  PR  subfactor  System  Integration  and  Software  (SIS)  rating. 

xs  :  PR  subfactor  Product  Support  (PS)  rating. 

xg  :  PR  subfactor  Program  Management  (PM)  rating. 

xio  :  Cost/Price  subfactor  Cost  Risk  (CR)  rating. 

Xu  :  Past  Performance  (PP)  rating. 

Objective  Function  (Linear  Model): 


Maximize  Overall  Score  =  (0.3  xi  +  0.25  x?  +  0.2  x.?  +0.15  x4  +  0. 1  xs )  0.25  + 

or  (0.35  xg  +  0.3  X7  +  0.20  xg  +  0. 1 5  xg)  0.25  + 

Minimize  0.25  x//  +  0. 1 25  *  90  + 

(0.40  xI0  +  0.60  *  39.8)  0.125 

Constraints: 


For  Mission  Capability; 
x i  >=  66  andx;  <=100 
X2  >=  26  and  X2  <=65 
X3  >=  66  andxi  <=100 
x4  >=  26  and  x4  <=65 
xj  >=  61  andx5  <=100 

For  Proposal  Risk; 
x<s>=61  andx,}<=100 
X7  >=  0  and  X7  <=60 
x,y  >=  6 1  and  x,y  <=  1 00 
xg>=6 1  andxp<=100 


(Subfactor  KSR  is  within  blue  color  rating  range) 
(Subfactor  SIS  is  within  green  color  rating  range) 
(Sub factor  PS  is  within  blue  color  rating  range) 
(Subfactor  PM  is  within  green  color  rating  range) 
(Subfactor  TMD  is  within  green  color  rating  range) 


(Subfactor  KSR  is  within  low  adjectival  rating  range) 
(Sub factor  SIS  is  within  moderate  adjectival  rating  range) 
(Sub factor  PS  is  within  low  adjectival  rating  range) 

(Sub factor  PM  is  within  low  adjectival  rating  range) 


For  Cost/Price; 

xio  >=  61  and  xw  <=100  (CR  is  within  low  adjectival  rating  range) 

For  Past  Performance; 

xu>=  31  and  X//  <=60  (PP  is  within  satisfactory  adjectival  rating  range) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  for  Weight  Changes 
(For  Boeing  for  Color  Rating  Method  in  Version  3) 


Decision  Variables: 

X]  :  Mission  Capability  (MC)  sub  factor  Key  System  Requirements  (KSR)  weight. 

x7  :  MC  subfactor  System  Integration  and  Software  (SIS)  weight. 

xj  :  MC  subfactor  Product  Support  (PS)  weight. 

x4  :  MC  subfactor  Program  Management  (PM)  weight. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  weight. 

X(,  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  weight. 

x7  :  PR  subfactor  System  Integration  and  Software  (SIS)  weight. 

x8  :  PR  subfactor  Product  Support  (PS)  weight. 

xg  :  PR  subfactor  Program  Management  (PM)  weight. 

xio  :  Cost/Price  subfactor  Life  Cycle  Cost  (LCC)  weight. 

xn  '■  Cost/Price  subfactor  Cost  Risk  (CR)  weight. 

x  12  :  Mission  Capability,  Proposal  Risk  and  Past  Performance  factors’  weight. 

Xu  :  Cost/Price  and  IFARA  factors’  weight. 


Objective  Function  (Non-linear  Model): 

Maximize  Overall  Score  =  [(87.5  Xj  +  62.5  x2+  87.5  xj+  62.5  x4+  83.3  X5)]  xn  + 
or  [(83.3  x6  +  50  x7  +  83.3  x8  +  83.3  x9)]  x12  + 

Minimize  58.35  xn  +  83.2  x»+  (50  Xu  +  39.12  xw)  X13 


Constraints: 


For  Mission  Capability; 
x 1  >x2 
x2>x3 
X3  >  x4 
X4  >  Xj 

xj>0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(PM  is  more  important  than  TMD) 
(all  of  the  weights  are  bigger  than  0) 


X1+X2  +  x  j  +  x4  +  xj  =  1  (total  of  weights  are  equal  to  1) 


For  Proposal  Risk; 

x6>x7 

Xy>X8 
X8>Xg 
Xg  >  0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(all  of  the  weights  are  bigger  than  0) 
x6  +  x7  +  x8  +  Xg  =  1  (total  of  weights  are  equal  to  1) 


For  Cost/Price; 

Xio  >  X11 
xn  >  0 


(LCC  is  more  important  than  CR) 
(all  of  the  weights  are  bigger  than  0) 


xio  +  xn=  1  (total  of  weights  are  equal  to  1) 


For  Factor  Level; 

Xi2>  X13  (MC,PR  and  PP  are  more  important  than  CR  and  IFARA) 

Xi3>  0  (all  of  the  weights  are  bigger  than  0) 

3  X/2  +  2x,j=  1  (total  of  weights  are  equal  to  1) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  for  Weight  Changes 
(For  Airbus  for  Color  Rating  Method  in  Version  3) 


Decision  Variables: 

X]  :  Mission  Capability  (MC)  sub  factor  Key  System  Requirements  (KSR)  weight. 
X2  :  MC  subfactor  System  Integration  and  Software  (SIS)  weight. 

x3  :  MC  subfactor  Product  Support  (PS)  weight. 

x4  :  MC  subfactor  Program  Management  (PM)  weight. 

X5  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  weight. 

X(,  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  weight. 

x7  :  PR  subfactor  System  Integration  and  Software  (SIS)  weight. 

x8  :  PR  subfactor  Product  Support  (PS)  weight. 

xg  :  PR  subfactor  Program  Management  (PM)  weight. 

xio  :  Cost/Price  subfactor  Life  Cycle  Cost  (LCC)  weight. 

xn  '■  Cost/Price  subfactor  Cost  Risk  (CR)  weight. 

x  12  :  Mission  Capability,  Proposal  Risk  and  Past  Performance  factors’  weight. 

xI3  :  Cost/Price  and  IFARA  factors’  weight. 

Objective  Function  (Non-linear  Model): 

Maximize  Overall  Score  =  [(87.5  xt  +  62.5  x2+  87.5  x3  +  62.5  x4+  83.3  x5)]  x12  + 
or  [(83.3  Xf,  +  50  X7  +  83.3  x8  +  83.3  xg)]  x32  + 

Minimize  58.35  x32  +  92  x73+  (83.3  Xu  +  39.8  xjo)  x33 


Constraints: 


For  Mission  Capability; 
x i  >x2 
x2>x3 

X3>X4 
X4  >  Xj 

Xj>0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(PM  is  more  important  than  TMD) 
(all  of  the  weights  are  bigger  than  0) 


xi+x2  +  x3  +  x4  +  xs  =  1  (total  of  weights  are  equal  to  1) 


For  Proposal  Risk; 

x6>x7 

x7>x8 

X8>Xg 

Xg  >  0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(all  of  the  weights  are  bigger  than  0) 
x6  +  x7  +  x8  +  Xg  =  1  (total  of  weights  are  equal  to  1) 


For  Cost/Price; 

Xio  >  X11 
xn  >  0 


(LCC  is  more  important  than  CR) 
(all  of  the  weights  are  bigger  than  0) 


xio  +  xn  =  1  (total  of  weights  are  equal  to  1) 


For  Factor  Level; 

Xi2>  xj3  (MC,PR  and  PP  are  more  important  than  CR  and  IFARA) 

xI3  >  0  (all  of  the  weights  are  bigger  than  0) 

3  x i2  +  2i/3=  1  (total  of  weights  are  equal  to  1) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  for  Weight  Changes 
(For  Boeing  for  WSM  Method  in  Version  3) 


Decision  Variables: 

X]  :  Mission  Capability  (MC)  sub  factor  Key  System  Requirements  (KSR)  weight. 
x7  :  MC  subfactor  System  Integration  and  Software  (SIS)  weight. 

x3  :  MC  subfactor  Product  Support  (PS)  weight. 

x4  :  MC  subfactor  Program  Management  (PM)  weight. 

X5  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  weight. 

X(,  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  weight. 

x7  :  PR  subfactor  System  Integration  and  Software  (SIS)  weight. 

x8  :  PR  subfactor  Product  Support  (PS)  weight. 

xg  :  PR  subfactor  Program  Management  (PM)  weight. 

xio  :  Cost/Price  subfactor  Life  Cycle  Cost  (LCC)  weight. 

xn  '■  Cost/Price  subfactor  Cost  Risk  (CR)  weight. 

x  12  :  Mission  Capability,  Proposal  Risk  and  Past  Performance  factors’  weight. 

xI3  :  Cost/Price  and  IFARA  factors’  weight. 

Objective  Function  (Non-linear  Model): 

Maximize  Overall  Score  =  [(93  X/  +  64  x2  +  92  x3  +  63  x4  +  94  x5)]  x12  + 
or  [(94  X6+  57  x7  +  97  x8  +  94xj>)]  Xn  + 

Minimize  59  x32  +  83.2  x33+  (58  Xu  +  39.12  xio )  x33 


Constraints: 


For  Mission  Capability; 
x i  >x2 
x2>x3 

X3>X4 
X4  >  Xj 

Xj>0 


For  Proposal  Risk; 

X6>X7 
Xy>X8 
X8>Xg 
Xg  >  0 


X6  +  X7  +  X8  +  Xg  =  1 


For  Cost/Price; 

Xio  >  X11 
xn  >  0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(PM  is  more  important  than  TMD) 
(all  of  the  weights  are  bigger  than  0) 


xi+x2  +  x3  +  x4  +  xs  =  1  (total  of  weights  are  equal  to  1) 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(all  of  the  weights  are  bigger  than  0) 
(total  of  weights  are  equal  to  1) 


(LCC  is  more  important  than  CR) 
(all  of  the  weights  are  bigger  than  0) 


Xio  +  Xn=  1  (total  of  weights  are  equal  to  1) 


For  Factor  Level; 

Xi2>  xj3  (MC,PR  and  PP  are  more  important  than  CR  and  IFARA) 

xI3  >  0  (all  of  the  weights  are  bigger  than  0) 

3  x12  +  2  xI3  =  1  (total  of  weights  are  equal  to  1) 
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Sensitivity  Analysis  Model  for  Best/Worst  Ratings  for  Weight  Changes 
(For  Airbus  for  WSM  Method  in  Version  3) 


Decision  Variables: 

X]  :  Mission  Capability  (MC)  sub  factor  Key  System  Requirements  (KSR)  weight. 

x7  :  MC  subfactor  System  Integration  and  Software  (SIS)  weight. 

x3  :  MC  subfactor  Product  Support  (PS)  weight. 

x4  :  MC  subfactor  Program  Management  (PM)  weight. 

xj  :  MC  subfactor  Technology  Maturity  and  Demonstration  (TMD)  weight. 

X(,  :  Proposal  Risk  (PR)  subfactor  Key  System  Requirements  (KSR)  weight. 

x7  :  PR  subfactor  System  Integration  and  Software  (SIS)  weight. 

x8  :  PR  subfactor  Product  Support  (PS)  weight. 

xg  :  PR  subfactor  Program  Management  (PM)  weight. 

xio  :  Cost/Price  subfactor  Life  Cycle  Cost  (LCC)  weight. 

xn  '■  Cost/Price  subfactor  Cost  Risk  (CR)  weight. 

x  12  :  Mission  Capability,  Proposal  Risk  and  Past  Performance  factors’  weight. 

xI3  :  Cost/Price  and  IFARA  factors’  weight. 

Objective  Function  (Non-linear  Model): 

Maximize  Overall  Score  =  [(8 1  x7  +  60  x2  +  90  xj  +  58  x4  +  85  Xj)]  x12  + 
or  [(85  jCtf  +  57  x7  +  91  x8  +  87  xg)\  x22  + 

Minimize  56  x32  +  92  x33+  (70  Xu  +  39.8  xjo)  x33 


Constraints: 


For  Mission  Capability; 
x i  >x2 
x2>x3 

X3>X4 
X4  >  Xj 

xj>0 


For  Proposal  Risk; 

x6>x7 

Xy>X8 
X8>Xg 
Xg  >  0 


X6  +  X7  +  X8  +  Xg  =  1 


For  Cost/Price; 

Xio  >  X11 
xn  >  0 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(PM  is  more  important  than  TMD) 
(all  of  the  weights  are  bigger  than  0) 


xi+x2  +  x3  +  x4  +  xj  =  1  (total  of  weights  are  equal  to  1) 


(KSR  is  more  important  than  SIS) 
(SIS  is  more  important  than  PS) 

(PS  is  more  important  than  PM) 

(all  of  the  weights  are  bigger  than  0) 
(total  of  weights  are  equal  to  1) 


(LCC  is  more  important  than  CR) 
(all  of  the  weights  are  bigger  than  0) 


Xio  +  Xn=  1  (total  of  weights  are  equal  to  1) 


For  Factor  Level; 

Xi2>  xj3  (MC,PR  and  PP  are  more  important  than  CR  and  IFARA) 

xI3  >  0  (all  of  the  weights  are  bigger  than  0) 

3  x i2  +  2x,3=  1  (total  of  weights  are  equal  to  1) 
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Microsoft  Excel  Snapshot  of  Sensitivity  Analysis  Model  for  Best  Rating  within  Color  Ranges  for  Boeing  in  Version  1 
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Microsoft  Excel  Snapshot  of  Sensitivity  Analysis  Model  for  Best  Rating  within  Color  Ranges  for  Boeing  in  Version  2 
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Microsoft  Excel  Snapshot  of  Sensitivity  Analysis  Model  for  Best  Rating  for  Weight  Changes  for  Boeing 

for  Color  Rating  Method  In  Version  3 
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Microsoft  Excel  Snapshot  of  Sensitivity  Analysis  Model  for  Best  Rating  for  Weight  Changes  for  Boeing 

for  WSM  Method  in  Version  3 
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Sensitivity  Analysis  Boeing’s  Best  Case  Scenario  Weights  in  Color  Rating  Method 

Applied  to  Airbus 
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Sensitivity  Analysis  Airbus’s  Best  Case  Scenario  Weights  in  Color  Rating  Method 

Applied  to  Boeing 
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Sensitivity  Analysis  1  Results  Summary  Table 
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Sensitivity  Analysis  2  Results  Summary  Table 
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Sensitivity  Analysis  3  Results  Summary  Table 
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